Method and apparatus for estimating delay and jitter between network routers coupled to background network

ABSTRACT

A delay measurement technique according to an embodiment according to the present invention is based on the precept, ascertained by the inventors, that a link between network nodes will often contribute to the delay encountered between several different pairs of network nodes. Such a technique identifies the path between each pair of nodes by a list of links that form the path. Paths that are orthogonal are treated as being necessary for describing the delays encountered between nodes, and, once the requisite set of orthogonal paths has been derived, all other paths can be described in terms of one or more of these orthogonal paths. Such a technique also lends itself to matrix representation of the paths, and the use of matrix manipulation techniques in deriving delay and jitter.

This application is a continuation of application Ser. No. 09/583,177,filed on May 30, 2000.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to computer networks, and, moreparticularly, to a method and apparatus for estimating a networkperformance metric, such as delay and jitter, between pairs of routersin a network.

2. Description of the Related Art

As computer networks grow, in terms of the number of routers containedtherein, the measurement of network performance metrics becomes ofincreasing importance. By measuring such metrics, network parameters canbe tuned in order to provide optimal performance. Moreover, thenetwork's architecture can be adjusted and growth planned to allow thenetwork to grow in a controllable fashion. One such metric is the delayexperienced by data packets flowing between certain of these routers(i.e., travel time between the routers). Another is the jitter, ordeviation in delay, experienced by such data packets. Thus, there is agrowing need to continuously monitor network delay and jitter betweenmultiple pairs of routers in a network such as an enterprise orservice-provider network. In service-provider networks, particularly,such performance monitoring is needed in order to verify service-levelagreements.

Unfortunately, current methods of monitoring are not as useful as mightbe desired. For example, one current method for monitoring network delayand jitter requires the measurement of delay and jitter between everyspecified pair of routers by exchanging probe packets between routers.As will be apparent to one of skill in the art, the number of pairs ofrouters that need to be monitored in such a scenario grows as aquadratic of N, where N is the number of network routers making up thenetwork. Thus, such a measurement technique involves calculations on theorder of N²(O(N²)).

Once generated, the measurement data is collected and processed. Themeasurement data can then be made available to other applications, suchas for the tuning of network parameters and the like. As can be seenfrom the technique's computational order (O(N²)), this measurementscheme does not scale well in large networks as the number of specifiedpairs of routers to be monitored increases dramatically. In such cases,the resulting network traffic due to probe packets can be large and,therefore, unsustainable as a result of the bandwidth consumed thereby.

This problem of scalability may be further compounded by the fact thatnetworks that will be deployed in the future will likely be diff-serv(DS) enabled. In such cases, delay and jitter characteristics must bemonitored for every DS-codepoint in use in the network. Diff-servenabled networks offer a range of data transfer services that aredifferentiated on the basis of performance experienced by packetsbelonging to a particular aggregated set of applications or flows. Anapplication requests a specific level of performance on apacket-by-packet basis, by marking the type-of-service (ToS) field ineach IP packet with a specific value, also called DS-codepoint. Thisvalue effectively specifies how an enterprise network or a serviceprovider network processes and forwards each packet along each hop inthe network.

What is therefore needed is a method and apparatus for the measurementof delays encountered by network traffic in traversing a network, thecomplexity of which preferably grows at a rate less than O(N²). Morepreferably, the complexity of the measurement scheme should grow at arate that increases at most linearly with the number of network routers(designated herein as N) and the number of network links (designatedherein as M).

SUMMARY OF THE INVENTION

The present invention, in various embodiments, addresses theaforementioned problems by providing techniques for measuring networkperformance metrics, such as delay encountered between nodes in anetwork. Advantageously, embodiments of the present invention allow suchmeasurements to be made without generating excessive network traffic,and are capable of accounting for delays both between nodes (the delayover inter-node links) and within nodes. Also advantageously,embodiments of the present invention are less demanding from ameasurement perspective, being on the order of N+M(O(N+M)).

A delay measurement technique according to an embodiment of the presentinvention is based on the precept, ascertained by the inventors, that alink between network nodes will often contribute to the delayencountered between several different pairs of network nodes. Such atechnique identifies the path between each pair of nodes by a list oflinks that form the path. Paths that are orthogonal are treated as beingnecessary for describing the delays encountered between nodes. Oncedelay and jitter over the requisite set of orthogonal paths have beenmeasured, delay over all other paths can be described in terms of thedelay and jitter encountered over one or more of these orthogonal paths.Such a technique also lends itself to a vector/matrix representation ofdelay and jitter over the paths, and the use of matrix manipulationtechniques in deriving delay and jitter.

The present invention allows such measurements to be made in severalsituations. Certain embodiments of the present invention allow delays tobe measured using a minimal set of paths. This technique can be extendedto the case where the number of paths used to represent the networkbeing analyzed approaches the number of paths in the network. Moreover,the techniques presented herein can be extended to the case where only aportion of the network's topology is known, for both the case of aminimal set of paths and a large number of paths.

The techniques described herein provide several advantages. Thesetechniques address the fundamental concern of a measurement scheme'sscalability in monitoring delay and jitter in increasingly largenetworks, with known or partly known topology and routing. Thetechniques result in a significant reduction in traffic overhead due toexchange of probe packets when compared to existing techniques. Thesavings can be especially significant in networks having a largerequirements set. Further, in a network employing TCP/IP protocols, themethods described herein separately accounts for IP-level transfer delayover a link and the delay in processing probe packets within a router(i.e., the delay experienced in generating and/or receiving suchpackets), thereby helping to isolate and identify links that are over orunder provisioned. Simplicity and ease of implementation are alsoattractive features of a technique according to the present invention.

In one embodiment of the present invention, a method of determining anetwork performance metric in a network is described. The networkincludes a number of network elements and a number of links. Each of thenetwork elements is coupled to at least one other of the networkelements by at least one of the links. The method includes measuring ameasured network performance metric between a first network element anda second network element of a network element pair in a first set ofnetwork element pairs. The first network element and the second networkelement of the network element pair in the first set of network elementpairs is included in the network elements. The measured networkperformance metric is such that a computed network performance metricbetween a first network element and a second network element of thenetwork elements can be computed using the measured network performancemetric.

In one aspect of this embodiment, the computed network performancemetric is computed. In this aspect, the computed network performancemetric is computed between a first network element and a second networkelement of a network element pair in a second set of network elementpairs, using the measured network performance metric (as noted above).The first network element and the second network element of the networkelement pair in the second set of network element pairs can, forexample, be included in the network elements. Moreover, the first set ofnetwork element pairs can, for example, be included in a second set ofnetwork element pairs.

In another embodiment of the present invention, a method of determininga network performance metric in a network is described. The networkincludes a number of network elements. Each one of the network elementsis coupled to at least one other network element by at least one of anumber of links. The method begins by identifying pairs of the networkelements as being in a first set of network element pairs. Next, a firstmatrix is generated from the first set of network element pairs. Eachrow in the first matrix corresponds to a corresponding network elementpair in the first set of network element pairs. The first matrixincludes independent rows and non-independent rows. A second set ofnetwork element pairs is then formed. The second set of network elementpairs contains independent network element pairs in the first set ofnetwork element pairs. Each one of the independent pairs of networkelement corresponds to one of the independent rows of the first matrix.Next, a measured network performance metric is measured between a firstnetwork element and a second network element of each network elementpair in the second set of network element pairs. Finally, a computednetwork performance metric between a first network element and a secondnetwork element of a remaining network element pair in the first set ofnetwork element pairs is computed using at least one of the measurednetwork performance metrics. The remaining network element paircorresponds to a non-independent row of the first matrix.

The foregoing is a summary and thus contains, by necessity,simplifications, generalizations and omissions of detail; consequently,those skilled in the art will appreciate that the summary isillustrative only and is not intended to be in any way limiting. As willalso be apparent to one of skill in the art, the operations disclosedherein may be implemented in a number of ways, and such changes andmodifications may be made without departing from this invention and itsbroader aspects. Other aspects, inventive features, and advantages ofthe present invention, as defined solely by the claims, will becomeapparent in the non-limiting detailed description set forth below.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention may be better understood, and its numerousobjects, features, and advantages made apparent to those skilled in theart by referencing the accompanying drawings.

FIG. 1A is a block diagram illustrating a computer system.

FIG. 1B is a block diagram illustrating the various sources of delayencountered by network traffic transiting a network.

FIG. 2 is a block diagram depicting a network in which a delaymeasurement technique according to an embodiment of the presentinvention can be practiced.

FIG. 3 is a flow diagram illustrating a process for computing theround-trip delay between pairs of routers according to an embodiment ofthe present invention.

FIG. 4 is a flow diagram illustrating a scheme according to anembodiment of the present invention capable of processing a requirementsset having a large number of nodes.

FIG. 5A is a flow diagram illustrating a method according to oneembodiment of the present invention for identifying a measurements setsuch that the resulting matrix has the maximum possible rank.

FIG. 5B is a flow diagram illustrating a method according to oneembodiment of the present invention for calculating delay and jitterbetween an arbitrarily specified pair of routers.

FIG. 6 is a block diagram illustrating an exemplary enterprise networkin which a method according to an embodiment of the present inventioncan be practiced.

FIG. 7 is a flow diagram illustrating the operations performed in aprocess of forming “clusters” (aggregations of network elements)according to an embodiment of the present invention.

FIG. 8A is a block diagram illustrating the topology of a synthesizednetwork and depicting an inferred backbone topology generated by amethod according to an embodiment of the present invention.

FIG. 8B is a block diagram illustrating a network and depicting aninferred backbone topology generated by a method according to anembodiment of the present invention.

FIG. 9 is a flow diagram illustrating a process according to anembodiment of the present invention that computes round-trip delay andjitter for router pairs contained in a specified requirements set usingmeasurements between router pairs in the measurements set.

FIG. 10A is a block diagram illustrating an exemplary network in which ameasurement technique according to an embodiment of the presentinvention can be practiced.

FIG. 10B is a block diagram illustrating an exemplary network afterbeing apportioned into clusters using a method according to anembodiment of the present invention.

FIG. 11 is a flow diagram illustrating the operations performed in theprocess of apportioning delay components according to an embodiment ofthe present invention.

FIG. 12 is a flow diagram illustrating the operations performed in theprocess of apportioning delay components according to an embodiment ofthe present invention, when the size of both a first cluster and asecond cluster are greater than or equal to two.

FIG. 13 is a flow diagram illustrating the operations performed in theprocess of apportioning delay components according to an embodiment ofthe present invention, when the size of a first cluster is greater thanor equal to two and the size of a second cluster is equal to one.

FIG. 14 is a flow diagram illustrating the operations performed in theprocess of apportioning delay components according to an embodiment ofthe present invention, when the size of both a first cluster and asecond cluster are equal to one.

FIG. 15 is a flow diagram illustrating the operations performed incomputing the measurements set for a given requirements set according toan embodiment of the present invention, when a maximal set ofindependent vectors from a corresponding set of row vectors is to befound.

FIG. 16 is a flow diagram illustrating the operations performed in aprocess of computing a measurements set according to an embodiment ofthe present invention.

The use of the same reference symbols in different drawings indicatessimilar or identical items.

DETAILED DESCRIPTION OF THE INVENTION

The following is intended to provide a detailed description of anexample of the invention and should not be taken to be limiting of theinvention itself. Rather, any number of variations may fall within thescope of the invention which is defined in the claims following thedescription.

Introduction

Embodiments of the measurement techniques described herein address thecentral issue of scalability. These methods allow (a)network performancemetrics, such as delay and jitter, to be measured between asignificantly smaller subset of pairs of routers, and (b) thecorresponding metrics for the remaining specified pairs of routers to becomputed based on these measurements. These methods include (a) theidentification of network paths for which measurements must beundertaken, and (b) the computation of the metrics for the remainingspecified pairs of routers for which measurements are not taken.

A definition of the following terms is helpful to the subsequentdiscussion regarding embodiments of the present invention:

-   -   1. The “requirements set” is defined herein as the specified        collection of pairs of routers for which delay and jitter        metrics are to be obtained.    -   2. The “measurements set” is defined herein as the set of router        pairs for which delay/jitter measurements are to be taken. A        fundamental property of the measurements set is that, based on        measurements thus obtained, one can accurately estimate        delay/jitter metrics for all pairs of routers specified in the        requirements set.

The methods described herein are based on a technique that divides theend-router-to-end-router path in terms of processing delay within thesource and destination routers and the IP-level transfer delay overintermediate link(s) (and router(s), if any) in the network path betweenthe end-routers. Moreover, techniques according to embodiments of thepresent invention can also be used in scenarios in which the internaltopology of the network being analyzed is unknown. Central to theseconcepts is (a) the identification of a minimal measurements set from agiven requirements set and (b) the estimation of delay and jitter forpairs of routers in the requirements set.

The methods described herein are such that only a minimal set of pairsof routers need be identified. This minimal set can be constructed suchthat the set's size is no more than N+M (the sum of the number ofnetwork routers and links). Even though the exact size of themeasurements set depends upon the topology, routing, and the specifiedrequirements set, a method according to the present invention results ina considerable savings in network traffic caused by the measurementprocess. To illustrate, in a network having 300 routers and 900 links,if the requirements set includes 5000 pairs of routers, the methodidentifies a “measurement set” consisting of at most 1200 pairs ofrouters (a saving of at least 76%). As another example, in a networkhaving 200 access-routers connected using a 10-node fully connected meshbackbone, the savings can be significantly higher depending upon thespecified requirements set. A more complete comparison is presentedsubsequently.

The methods described herein can be extended to handle a requirementsset that includes a significant fraction of all pairs of routers. Tocope with such cases, the method can be made to directly identify themeasurements set and the estimation scheme, without having to processeach pair of routers specified in the requirements set. It will be notedthat, although the techniques disclosed herein are generally applicableto various network protocols and topologies, these techniques arediscussed herein in terms of TCP/IP protocols.

Other than resulting in considerably reduced network traffic due toprobe packets, a monitoring scheme based on this method enables one toestimate the IP-level transfer delay over each router-to-router link inthe network. Such an estimate factors out the delay in processing(generating/receiving) probe packets within a router. This is desirablebecause the delay in processing probe packets can be significant as aresult of probes being generated and received within the applicationlayer. It will be apparent to one skilled in the art that an assessmentof IP-level transfer delay over each link assists a networkadministrator in identifying congested links in a functioning network.

The methods described above are applicable to networks for which (a)topology and routing information is available, and (b) changes intopology and routing are infrequent. These methods may also be extendedto apply to networks that are built around a network backbone, thetopology and routing of which are unknown.

Such a method directly provides a measurements set and a scheme forestimating delay and jitter between the routers on such a network. As aresult, one is also able to apportion the delay-components between theenterprise network and the backbone network.

The techniques described herein provide several advantages. Thesetechniques address the fundamental concern of a measurement scheme'sscalability in monitoring delay and jitter in increasingly largenetworks, with known or partly known topology and routing. Thetechniques result in a significant reduction in network traffic overheaddue to exchange of probe packets when compared to existing techniques.The savings can be especially significant in networks having a largerequirements set. Further, the methods described herein separatelyaccount for IP-level transfer delay over a link and the delay inprocessing probe packets within a router (i.e., the delay experienced ingenerating and/or receiving such packets), thereby helping to isolateand identify links that are over or under provisioned. Simplicity andease of implementation are also attractive feature of these techniques.

The following discussion of measurement schemes according to variousembodiments of the present invention is divided into four primarysections. First is a description of measurement technique that uses aminimal number of router pairs to measure delay and jitter in anenterprise network having a known topology. Next, a scheme for takingmeasurements using a large number of router pairs to measure delay andjitter in an enterprise network having a known topology is discussed.The estimation of delay and jitter in an enterprise network coupled to abackbone having an unknown topology is then discussed. In effect, thissituation deals with a portion of the network being unknown (e.g., aservice provider backbone) and making the requisite estimation using aminimal measurements set. This discussion includes making suchestimations with regard to virtual links between routers within thebackbone. Finally, estimation of delay and jitter in an enterprisenetwork coupled to a backbone having an unknown topology and using alarge number of router pairs is examined.

I. Measurements Using a Minimal Number of Router Pairs in a NetworkHaving a Known Topology

An active network monitoring tool (referred to generically herein as anANMT) can be used to measure round-trip delay, delay variation (jitter)and packet loss between a specified set of pairs of routers (or anetwork switch with a router switch module or an attached one-armrouter) in a large diff-serv enabled enterprise network. By design, ANMTsoftware running as an application on a router (e.g., a source router),sends a sequence of user datagram protocol (UDP) packets (also known asprobe packets) to a specified port on a remote device (e.g., adestination router). Based on information contained in UDP responsesreceived from the destination, the ANMT software run on the sourcecomputes various statistics, such as average round-trip delay, standarddeviation of delay jitter, and average fraction of packets dropped. AnANMT may also be used to test TCP-level connectivity or to estimateone-way packet loss and delay-variation. Additionally, an ANMT allowsmonitoring of performance between devices in a network which offersdifferentiated services.

A method according to one embodiment of the present invention permitsthe use of a network measurement tool (e.g., an ANMT) to measureround-trip delay and jitter between a much smaller subset of pairs ofrouters (represented herein by the notation Ω), in order to estimatedelay and jitter between pairs of routers in a specified set(represented herein by the notation Φ). This subset Ω includes at most(N+M) pairs of routers, where N is the number of enterprise routers, andM is the number of links. This is a significant improvement over thecomplexity of measuring delay between every pair of routers in the set Φusing a tool such as an ANMT. It will be noted that the number of pairsof routers in Φ may be as large as N(N+1)/2, or O(N²). Second, thetechniques discussed herein recognize that delay encountered ingenerating, receiving and processing probe packets within the transportand application layers in a router may be significant.

FIG. 1A depicts a block diagram of a computer system 10 suitable forimplementing the present invention, and exemplary of one or more ofclient terminals 112(1)-(N). Computer system 10 includes a bus 12 whichinterconnects major subsystems of computer system 10 such as a centralprocessor 14, a system memory 16 (typically RAM, but which may alsoinclude ROM, flash RAM, or the like), an input/output controller 18, anexternal audio device such as a speaker system 20 via an audio outputinterface 22, an external device such as a display screen 24 via displayadapter 26, serial ports 28 and 30, a keyboard 32 (interfaced with akeyboard controller 33), a storage interface 34, a floppy disk drive 36operative to receive a floppy disk 38, and a CD-ROM drive 40 operativeto receive a CD-ROM 42. Also included are a mouse 46 (or otherpoint-and-click device, coupled to bus 12 via serial port 28), a modem47 (coupled to bus 12 via serial port 30) and a network interface 48(coupled directly to bus 12).

Bus 12 allows data communication between central processor 14 and systemmemory 16, which may include both read only memory (ROM) or flash memory(neither shown), and random access memory (RAM) (not shown), aspreviously noted. The RAM is generally the main memory into which theoperating system and application programs are loaded and typicallyaffords at least 16 megabytes of memory space. The ROM or flash memorymay contain, among other code, the Basic Input-Output system (BIOS)which controls basic hardware operation such as the interaction withperipheral components. Applications resident with computer system aregenerally stored on and accessed via a computer readable medium, such asa hard disk drive (e.g., fixed disk 44), an optical drive (e.g., CD-ROMdrive 40), floppy disk unit 36 or other storage medium. Additionally,applications may be in the form of electronic signals modulated inaccordance with the application and data communication technology whenaccessed via network modem 47 or interface 48.

Storage interface 34, as with the other storage intefaces of computersystem 10, may connect to a standard computer readable medium forstorage and/or retrieval of information, such as a fixed disk drive 44.Fixed disk drive 44 may be a part of computer system 10 or may beseparate and accessed through other interface systems. Many otherdevices can be connected such as a mouse 46 connected to bus 12 viaserial port 28, a modem 47 connected to bus 12 via serial port 30 and anetwork interface 48 connected directly to bus 12. Modem 47 may providea direct connection to a remote server via a telephone link or to theInternet via an internet service provider (ISP). Network interface 48may provide a direct connection to a remote server via a direct networklink to the Internet via a POP (point of presence).

Many other devices or subsystems (not shown) may be connected in asimilar manner. Conversely, it is not necessary for all of the devicesshown in Fig. to be present to practice the present invention. Thedevices and subsystems may be interconnected in different ways from thatshown in FIG. 1A. The operation of a computer system such as that shownin FIG. 1A is readily known in the art and is not discussed in detail inthis application. Code to implement the present invention may beoperably disposed or stored in computer-readable storage media such asone or more of system memory 16, fixed disk 44, CD-ROM 42, or floppydisk 38.

It will be noted that the variable identifier “N” is used in severalinstances in FIG. 1A to more simply designate the final element (e.g.,servers 110(1)-(N) and client terminals 112(1)-(N)) of a series ofrelated or similar elements (e.g., servers and client terminals). Therepeated use of such variable identifiers is not meant to imply acorrelation between the sizes of such series of elements, although suchcorrelation may exist. The use of such variable identifiers does notrequire that each series of elements has the same number of elements asanother series delimited by the same variable identifier. Rather, ineach instance of use, the variable identified by “N” may hold the sameor a different value than other instances of the same variableidentifier.

Moreover, regarding the signals described herein, those skilled in theart will recognize that a signal may be directly transmitted from afirst block to a second block, or a signal may be modified (e.g.,amplified, attenuated, delayed, latched, buffered, inverted, filtered orotherwise modified) between the blocks. Although the signals of theabove described embodiment are characterized as transmitted from oneblock to the next, other embodiments of the present invention mayinclude modified signals in place of such directly transmitted signalsas long as the informational and/or functional aspect of the signal istransmitted between blocks. To some extent, a signal input at a secondblock may be conceptualized as a second signal derived from a firstsignal output from a first block due to physical limitations of thecircuitry involved (e.g., there will inevitably be some attenuation anddelay). Therefore, as used herein, a second signal derived from a firstsignal includes the first signal or any modifications to the firstsignal, whether due to circuit limitations or due to passage throughother circuit elements which do not change the informational and/orfinal functional aspect of the first signal.

The foregoing described embodiment wherein the different components arecontained within different other components (e.g., the various elementsshown as components of computer system 10). It is to be understood thatsuch depicted architectures are merely exemplary, and that in fact manyother architectures can be implemented which achieve the samefunctionality. In an abstract, but still definite sense, any arrangementof components to achieve the same functionality is effectively“associated” such that the desired functionality is achieved. Hence, anytwo components herein combined to achieve a particular functionality canbe seen as “associated with” each other such that the desiredfunctionality is achieved, irrespective of architectures or intermedialcomponents. Likewise, any two components so associated can also beviewed as being “operably connected”, or “operably coupled”, to eachother to achieve the desired functionality.

FIG. 1B depicts a block diagram illustrating the various sources ofdelay encountered by network traffic transiting a network 100. While amethod according to an embodiment of the present invention may bepracticed using any one of a number of protocols, FIG. 1B is illustratedin terms of a TCP/IP network protocol. Delay can be apportioned intomonitor application response delay (exemplified in FIG. 1B by monitorapplication response delays 105 and 110) and network layer transferdelays (exemplified in FIG. 1B by network layer transfer delays115(1)-(N)) over an (end-to-end) network path 120 between a sourcerouter 125 and a destination router 130. In transiting network 100,network path 120 may also transit one or more intermediate routers(exemplified in FIG. 1B by an intermediate router 135). Source router125, intermediate router 135 and destination router 130 are alsodesignated as R₁, R₂, and R_(N), indicating the progression of networktraffic transiting network 100. In a TCP/IP environment such as that ofnetwork 100, monitor application delays 105 and 110 include,respectively, delays attributable to a network monitor 140, andtransport layer 145, a transport layer 146 and a network monitor 141. Ina similar fashion, network layer transfer delays 115(1)-(N) includedelays attributable to moving IP packets through and between IP layers150, 151 and 152, intermediate layers 155, 156 and 157, and physicallayers 160, 161 and 162, in the manner shown in FIG. 1B. As can be seenin FIG. 1B, the two principle types of delay components contributing tothe round-trip delay between a pair of routers such as source router 125and destination router 130 when measured using an ANMT can be definedas:

-   -   d_(j), j=1, 2, . . . , M, the round-trip IP-level delay in        transferring an IP packet over a link, L_(j), between two        adjacent routers; and    -   s_(i), i=1, 2, . . . , N, the time to generate, receive and        process a probe packet within transport and application layers        in a router, R_(i).

The symbols used herein in describing techniques according to variousembodiments of the present invention for estimating round-trip delay andjitter between a specified set of pairs of routers are now presented.

FIG. 2 is a block diagram depicting a network 200 that illustrates thecomponents of the delay existing between various pairs of routers thatmake up network 200. As is illustrated in FIG. 2, network 200 includes anumber of routers (exemplified in FIG. 2 by routers 210(1)-(6)) whichare coupled to one another by a number of interconnecting links(exemplified in FIG. 2 by links 220(1)-(9)). In order to facilitate thefollowing discussions, routers 210(1)-(6) are also designated R₁-R₆,respectively. In similar fashion, links 220(1)-(9) are referred to usingL₁-L₉, respectively. In the general sense, network 200 can be describedas including N routers, R_(i), i=1, 2, . . . , N, and M interconnectinglinks, L_(j), j=1, 2, . . . , M. A route between a given pair ofrouters, π_(k)=(R_(i), R_(j)), is specified as an ordered list ofintermediate links that network traffic between R_(i) and R_(j)traverses:

 link_path(R _(i) , R _(j))=<L _(p) , L _(q) , . . . , L _(r)>  (1)

That is, network traffic moving from source router R_(i) to destinationrouter R_(j) traverse the links, L_(p), L_(q), . . . , and L_(r) in thatorder, and vice-versa, in the other direction.

For a given DS codepoint, the routes between routers, R_(i) and R_(j),are given in Table 1. It will be noted that the routes in Table 1 arespecified for a given DS codepoint, and in terms of intermediate links.

TABLE 1 Routing table for network given in FIG. 2. R₁ R₂ R₃ R₄ R₅ R₆ R₁— L₁ L₁, L₂ L₃ L₃, L₈ L₃, L₈, L₉ R₂ — L₂ L₄ L₅ L₂, L₇ R₃ — L₆, L₈ L₆ L₇R₄ — L₈ L₈, L₉ R₅ — L₉ R₆ —

As noted, the two principal types of delay components that contribute tothe round-trip delay between a pair of routers when measured using anANMT can be defined as d_(j), j=1, 2, . . . , M (the round-trip IP-leveldelay in transferring an IP packet over a link, L_(j)), and s_(i), i=1,2, . . . , N (the time to generate, receive and process a probe packetwithin transport and application layers in a router, R_(i)).

As a result, the round-trip delay between any given pair of routers,R_(i) and R_(j), is given by:Delay(R _(i) , R _(j))=s _(i) +d _(p) +d _(q) + . . . +d _(r) +s_(j)  (2)assuming that the route between R_(i) and R_(j) is known and specifiedin terms of the intermediate links. That is,link_path(R _(i) , R _(j))=<L _(p) , L _(q) , . . . , L _(r)>  (3)When used between routers, R_(i) and R_(j), a delay monitoring tool suchas an ANMT measures Delay(R_(i), R_(j)) (which may include measurementerror). The requirements set and measurements set can now be definedusing the framework set forth above.

The user-specified requirements set of P number of pairs of routersbetween which round-trip delay is to be measured is given byΦ={π_(k) , k=1, 2, . . . , P}  (4)whereπ_(k)=(R _(i) , R _(j))  (5)and soΦ={(R _(i) , R _(j)), i,j=1, 2, . . . , P}  (6)Normally, the subset will only be a small fraction of all possible pairsof routers. As noted, because round-trip delay or jitter is monitored,it is immaterial whether the pair of routers is specified asπ_(k)=(R_(i), R_(j)) or as π_(k)=(R_(i), R_(j)).

The estimate is based on measurements between a subset of pairs ofrouters (and indeed what can be a minimal subset of router pairs), whichmay be referred to by Ω. That is, for every (R_(i), R_(j),)εΦ, the delaybetween the pair of routers (R_(i), R_(j)) can be computed (orestimated) in terms of measured delay between one or more pairs ofrouters {(R_(p), R_(q))}εΩ. The set, Ω, is also referred to herein asthe measurements set. A method according to the present inventionidentifies the measurements set, Ω, for a given requirements set, Φ.

Determination of the Measurements Set

A method according to an embodiment of the present invention is nowdescribed that is capable of identifying the collection of pairs ofrouters, or Ω, such that the delay between a given pair of routers(R_(i), R_(j),)εΦ can be computed from measurements between a subset ofpairs of routers (R_(p), R_(q),)εΩ.

If the number of pairs of routers in Φ is defined as P (or, P=|Φ|), theround-trip delay between the k^(th) pair of routers, (R_(i), R_(j),)εΦis denoted by z_(k)=Delay(R_(i), R_(j)). In view of the definition ofdelay components, it can be seen that, for every k=1, 2, . . . , P:z _(k)=Delay(R _(i) , R _(j))=s _(i) +d _(p) +d _(q) + . . . +d _(r) +s_(j)  (7a)assuming that the route between R_(i) and R_(j) is known and given bylink_path(R _(i) , R _(j))=<L _(p) , L _(q) , . . . , L _(r)>  (7b)Using a vector notation, z_(k)=Delay(R_(i), R_(j)) may be re-written as$\quad\begin{matrix}{z_{k} = {{Delay}\left( {R_{i},R_{j}} \right)}} & {\quad\left( {8a} \right)} \\{= {s_{i} + d_{p} + d_{q} + \ldots + d_{r} + s_{j}}} & {\quad\left( {8b} \right)} \\{= F_{x}} & {\quad\left( {8c} \right)}\end{matrix}$And further, the set of z_(k) (k=1, . . . , P) can be re-writtenz=[z ₁ z ₂ . . . z _(P)]^(T)is a column vector of size P. Further,F _(k)=[0 . . . 0 1 0 . . . 0 1 0 . . . 0 1 0 . . . 0 1 0 . . . 0 1 0 .. . 0]  (9)is a row vector of size N+M, and the 1's in F_(k) appear in columns p,q, . . . , r, and in columns M+i and M+j.

The ordered collection of delay components can be re-written in the formof a vector of size N+M (the delay-components vector).x=[d ₁ , d ₂ , . . . , d _(M) , s ₁ , s ₂ , . . . , s _(N)]^(T)  (10)or $\quad\begin{matrix}\begin{matrix}{x = \begin{bmatrix}d_{1} & \quad\end{bmatrix}} \\{\begin{bmatrix}d_{2} & \quad\end{bmatrix}} \\{\begin{bmatrix}\ldots & \quad\end{bmatrix}} \\{\begin{bmatrix}d_{M} & \quad\end{bmatrix}} \\{\begin{bmatrix}s_{1} & \quad\end{bmatrix}} \\{\begin{bmatrix}s_{2} & \quad\end{bmatrix}} \\{\begin{bmatrix}\ldots & \quad\end{bmatrix}} \\{\begin{bmatrix}s_{N} & \quad\end{bmatrix}}\end{matrix} & (11)\end{matrix}$

As noted, the round-trip delay between the specified pairs of routers inΦ={π_(k)=(R_(i), R_(j)), k=1, 2, . . . , P}, where z_(k)=F_(k)x and k=1,2, . . . , P, can be re-written as a vector equationz=Fx  (13)where $\quad\begin{matrix}{\begin{matrix}{z = \begin{bmatrix}z_{1} & \quad\end{bmatrix}} \\{\begin{bmatrix}z_{2} & \quad\end{bmatrix}} \\{\begin{bmatrix}\ldots & \quad\end{bmatrix}} \\{\begin{bmatrix}z_{p} & \quad\end{bmatrix}}\end{matrix},\begin{matrix}{F = \begin{bmatrix}F_{1} & \quad\end{bmatrix}} \\{\begin{bmatrix}F_{2} & \quad\end{bmatrix}} \\{\begin{bmatrix}\ldots & \quad\end{bmatrix}} \\{\begin{bmatrix}F_{p} & \quad\end{bmatrix}}\end{matrix}} & (14)\end{matrix}$The P×(N+M) matrix, F, plays an important role in determining the subsetof pairs of routers between which delay measurements are necessary.

Consider the row vectors, F_(k), k=1, 2, . . . , P and let Q be themaximum number of independent row vectors. As can be seen, Q is equal tothe rank of matrix F (i.e., Rank(F)). Without loss of generality, F₁ F₂. . . F_(Q) be the independent row vectors of matrix F. This is withoutloss of generality because the row vectors F_(k) in F (and similarly thecorresponding pairs of routers in Φ) can be re-arranged, if necessary.Then, every row vector F_(k), k=Q+1, Q+2, . . . , P, can be expressed asa linear combination of row vectors, F_(k), k=1, 2, . . . , Q.

The relationship between the linearly independent and linearly dependentrows of F is now examined. If F₁, F₂, . . . , F_(Q) is a maximal set oflinearly independent rows of F, then row vectors F_(Q+1), F_(Q+2), . . ., F_(P) can be expressed as a linear combination of F₁, F₂, . . . ,F_(Q). In other words, F_(k) can be expressed in terms of:

 F _(k)=Σ_(i=1, . . . , Q)(α_(k,i) F _(i)), k=Q+1, Q+2, . . . , P  (15)

The constants α_(k,i) may be re-organized in the form of a row vector(of size Q) in the following manner:α_(k)=[α_(k,1), α_(k,2), . . . , α_(k,Q) ], k=Q+1, Q+2, . . . , P  (16)The values of α_(K) may be aggregated to form the P×Q matrix, A, whichcan be defined as: $\quad\begin{matrix}\begin{matrix}{A = \begin{bmatrix}I & \quad\end{bmatrix}} \\{\begin{bmatrix}\alpha_{Q + 1} & \quad\end{bmatrix}} \\{\begin{bmatrix}\alpha_{Q + 2} & \quad\end{bmatrix}} \\{\begin{bmatrix}\ldots & \quad\end{bmatrix}} \\{\begin{bmatrix}\alpha_{p} & \quad\end{bmatrix}}\end{matrix} & (17)\end{matrix}$where the matrix, I, is a Q×Q matrix. Because delay (z_(k)) is a randomvariable, delay-jitter between pair of routers (π_(k)=(R_(i), R_(j)))can, be defined as follows:Delay-Jitter(R _(i) , R _(j))=√E{(z _(k) −E{z _(k)})²}  (18)where E{.} is the expectation operation.

In regard to a measurement method according to certain embodiments ofthe present invention, certain assumptions concerning the network beinganalyzed are now summarized. The network is assumed to have thefollowing characteristics:

-   For the embodiments discussed above, it is assumed that the    network's topology is known. It will be noted that, because a route    between a pair of routers is specified in terms of links, there is    no constraint on the existence of parallel links between a pair of    enterprise routers.-   Further, it is preferable that the topology not change frequently.    If the topology of the network changes, then the procedure described    herein should be carried out afresh.-   For each DS codepoint, the route between every pair of enterprise    routers should be known.-   The route between every pair of enterprise routers for a given DS    codepoint is symmetric.-   The delay and jitter characteristics between a pair of routers may    be different in the two directions.

In other words, certain of the procedures described herein arepreferably used together with techniques that monitor changes intopology and in routing in measuring delay and jitter in the network.Once a change in topology or in routing is detected by these techniques,the method according to an embodiment of the present invention should beperformed once again.

As a consequence of the foregoing manipulations, certain results can bededuced. As outlined below, these results include:

-   -   1. the fact that, for every (R_(i), R_(j))εΦ, Delay(R_(i),        R_(j)) can be expressed as a linear combination of one or more        Delay(R_(p), R_(q)), (R_(p), R_(q))εΩ; and    -   2. the fact that the number of router-to-router delay        measurements required to estimate delay between an arbitrarily        specified subset of pairs of routers is at most M+N.        These results are set forth in detail below.        First Result

The first result is determined as follows. Consider the matrix F=[F₁ F₂. . . F_(P)]^(T), corresponding to the given set of router pairs in therequirements set, Φ. Then, the minimal subset of pairs of routers, Ω,between which delay must be measured is given by the collection of allpairs of routers, (R_(p), R_(q)), that correspond to the maximal set ofindependent rows of F, with respect to {F_(k), k=1, 2, . . . , Q}. Inother words, the measurements set can be represented as:Ω={(R _(p) , R _(q)), such that Delay(R _(p) , R _(q))=F _(k) x, k=1, 2,. . . , Q}  (19)As a consequence, for every (R_(i), R_(j))εΦ, Delay(R_(i), R_(j)) can beexpressed as a linear combination of one or more Delay(R_(p), R_(q)),where (R_(p), R_(q))εΩ.

The above result is illustrated by the following example. Consider thenetwork of FIG. 2, where N=6, M=9. If the requirements set, Φ={(R₃, R₄)(R₃, R₅) (R₄ R₆) (R₅, R₆)}, then from routing table given in Table 1 fora given DS codepoint:z ₁=Delay(R ₃ , R ₄)=s ₃ +d ₆ +d ₈ +s ₄z ₂=Delay(R ₃ , R ₅)=s ₃ +d ₆ +s ₅z ₃=Delay(R ₄ , R ₆)=s ₄ +d ₈ +d ₉ +s ₆z ₄=Delay(R ₅ , R ₆)=s ₅ +d ₉ +s ₆That is, z=[z ₁ z ₂ z ₃ z ₄]^(T) =Fx  (20)wherex=[d ₁ , d ₂ , . . . , d ₉ , s ₁ , s ₂ , . . . , s ₆]^(T)  (21)

The F matrix is a 4×15 matrix of the form:

D₁ D₂ D₃ D₄ D₅ D₆ D₇ D₈ D₉ S₁ S₂ S₃ S₄ S₅ S₆ z₁ 1 1 1 1 z₂ 1 1 1 z₃ 1 11 1 z₄ 1 1 1It is verified subsequently herein that Rank(F)=3, and that z₂, z₃ andz₄ are independent row vectors, and that z₁=z₂+z₃−z₄. Thus, themeasurements set, Ω={(R₃, R₅), (R₄ R₆), (R₅, R₆)}.

Using a different requirements set for network 200, if$\quad\begin{matrix}\begin{matrix}{\Phi = \left\{ {\left( {R_{1},R_{2}} \right),\left( {R_{2},R_{3}} \right),\left( {R_{1},R_{4}} \right),\left( {R_{2},R_{4}} \right),\left( {R_{2},R_{5}} \right),} \right.} \\{\left( {R_{3},R_{5}} \right),\left( {R_{3},R_{6}} \right),\left( {R_{4},R_{5}} \right),\left( {R_{5},R_{6}} \right),\left( {R_{1},R_{3}} \right),} \\\left. {\left( {R_{2},R_{6}} \right),\left( {R_{1},R_{5}} \right),\left( {R_{4},R_{6}} \right),\left( {R_{1},R_{6}} \right)} \right\}\end{matrix} & (22)\end{matrix}$the resulting matrix F is

D₁ D₂ D₃ D₄ D₅ D₆ D₇ D₈ D₉ S₁ S₂ S₃ S₄ S₅ S₆ z₁  1 1 1 z₂  1 1 1 z₃  1 11 z₄  1 1 1 z₅  1 1 1 z₆  1 1 1 z₇  1 1 1 z₈  1 1 1 z₉  1 1 1 z₁₀ 1 1 11 z₁₁ 1 1 1 1 z₁₂ 1 1 1 1 z₁₃ 1 1 1 1 z₁₄ 1 1 1 1 1

It can be shown that the matrix F has rank 13, and that the first 13rows of F are independent. Further, z₁₄=z₁₂+z₁₃−z₈. Thus Ω{(R₁, R₂),(R₂, R₃), (R₁, R₄), (R₂, R₄), (R₂, R₅), (R₃, R₅), (R₃, R₆), (R₄, R₅),(R₅, R₆), (R₁, R₃), (R₂, R₆), (R₁, R₅), (R₄, R₆)}. It will be noted thatQ is also the subset of measurements required to estimate the delaybetween an arbitrary pair of routers. The second result is now examined.

Second Result

The minimal subset of pairs of routers, between which delay should bemeasured, so that delay between every other pair of routers can beobtained, can also be determined. The number of router-to-router delaymeasurements required to estimate delay between an arbitrarily specifiedsubset of pairs of routers is at most M+N. In other words:Q=|Ω|=Rank(F)≦min(|Φ|, N+M)  (23)This is so, because the matrix, F, is a |Φ|×(M+N), the rank of which,with respect to Q, is at most |Φ| or (M+N), whichever is smaller. As isevident from the approximate minimum savings in network traffic setforth in Table 2 below, the advantages provided by the foregoing resultscan be substantial.

TABLE 2 Approximate minimum saving in terms probe packet traffic. N + %max- % N N*(N − 1)/2 M M monitored P Q savings 20 190 60 80 5 9 9 — 1019 19 — 20 38 38 — 40 76 76 — 80 152 80 47.37 100 120 5 9 9 — 10 19 19 —20 38 38 — 40 76 76 — 80 152 120 21.05 80 3160 240 320 5 158 158 — 10316 316 — 20 632 320 49.37 40 1264 320 74.68 80 2528 320 87.34 400 480 5158 158 — 10 316 316 — 20 632 480 24.05 40 1264 480 62.03 80 2528 48081.01 320 51040 960 1280 5 2552 1280 49.84 10 5104 1280 74.92 20 102081280 87.46 40 20416 1280 93.73 80 40832 1280 96.87 1600 1920 5 2552 192024.76 10 5104 1920 62.38 20 10208 1920 81.19 40 20416 1920 90.60 8040832 1920 95.30The theoretical underpinnings of the foregoing results are now examined.Theoretical Underpinnings of the Measurement Scheme

A measurement scheme according to certain embodiments of the presentinvention, as described herein, makes Q measurements between thecollection of Q pairs of routers, specified by Ω. It will be noted thatQ=|Ω|, and that Ω is the collection of pairs of routers, (R_(p), R_(q)),that correspond to the maximal set of independent rows of F, withrespect to {F_(k), k=1, 2, . . . , Q} (i.e., the measurements set).

Network analysis agents (e.g., ANMTs) can be executed, for example, astransport applications in two routers, R_(i) and R_(j), to measureround-trip delay and jitter between R_(i) and R_(j). Such a measurement,denoted y_(k) yields:y _(k)=Delay(R _(i) , R _(j))+v _(k) =s _(i) +d _(p) +d _(q) + . . . +d_(r) ,+s _(j) +v _(k)  (24)

It is assumed that the intermediate links between the two routers areL_(p), L_(q), . . . , L_(r), that the measurement error is v_(k). Asnoted, processing delays encountered by probe packets within source anddestination routers, R_(i) and R_(j), respectively, are defined as s_(i)and s_(j), and d_(p), d_(q), . . . , d_(r) are round-trip IP-layertransfer delays encountered by probe packets over intermediate linksalong the routelink_route(R _(i) , R _(j))=<L _(p) , L _(q) , . . . , L _(r)>  (25)The k^(th) measurement, y_(k), k=1, 2, . . . , Q, of round-trip delaybetween the k^(th) pair of routers, R_(i) and R_(j), in Ω, may bere-written as: $\quad\begin{matrix}\begin{matrix}{y_{k} = {{{{Delay}\quad\left( {R_{i},R_{j}} \right)} + v_{k}} = {s_{i} + d_{p} + d_{q} + \ldots + d_{r} + s_{j} + v_{k}}}} \\{= {{F_{x}\quad x} + v_{k}}}\end{matrix} & (26)\end{matrix}$where the row vector (of size M+N)F _(k)=[0 . . . 0 1 0 . . . 0 1 0 . . . 0 1 0 . . . 0 1 0 . . . 0 1 0 .. . 0]  (27)and the 1's in F_(k) appear in columns p, q, . . . , r, and in columnsM+i and M+j. The following can be defined:y=[y ₁ y ₂ . . . y _(Q)]^(T)  (28a)x=[d ₁ , d ₂ , . . . , d _(M) , s ₁ , s ₂ , . . . , s _(N)]^(T)  (28b)v=[v ₁ v ₂ . . . v _(Q)]^(T)  (28c)where vector y, of size Q, represents the collection of measurements,vector x, of size M+N, is the collection of delay components and vectorv, of size Q, is the measurement error in y. The measurements, y_(k),k=1, 2, . . . , Q, taken together, may now be re-written in the form ofa vector equationy=Hx+v  (29)where the Q×(M+N) matrix H=[F₁ ^(T) F₂ ^(T) . . . F_(Q) ^(T)]^(T).Because the k measurements correspond to pairs of routers identified byΩ, the row vectors, F_(k), k=1, 2, . . . , Q, are all independent. As aconsequence, matrix H has full rank with respect to Q.

Consider, for example, the network given in FIG. 2, where N=6 and M=9.The routes between pairs of routers, R_(i) and R_(j), are given inTable 1. Let Ω={(R₃, R₅) (R₄ R₆) (R₅, R₆)} be the specified subset ofpairs between which delay is measured. From Table 1,y ₁=Delay(R ₃ , R ₅)+v ₁ =s ₃ +d ₆ +s ₅ +v ₁  (30a)y ₂=Delay(R ₄ , R ₆)+v ₂ =s ₄ +d ₈ +d ₉ +s ₆ +v ₂  (30b)y ₃=Delay(R ₅ , R ₆)+v ₃ =s ₅ +d ₉ +s ₆ +v ₃  (30c)That is,y=Hx+v  (31)wherey=[y ₁ y ₂ y ₃]^(T)  (32a)x=[d ₁ , d ₂ , . . . , d ₉ , s ₁ , s ₂ , . . . , s ₆]^(T)  (32b)v=[v ₁ v ₂ v ₃]^(T)  (32c)and 3×15 matrix H is:

D₁ D₂ D₃ D₄ D₅ D₆ D₇ D₈ D₉ S₁ S₂ S₃ S₄ S₅ S₆ Y₁ 1 1 1 Y₂ 1 1 1 1 Y₃ 1 11Calculating Delay within a Requirements Set Using a Measurements Set

In this section, a methodology for computing the round-trip delaybetween every pair of routers within a given requirements set, (R_(i),R_(j))εΦ, based on measurements between pairs of routers identified bythe measurements set, Ω (a subset of the requirements set), isdescribed. This approach is particularly useful when measurement errorsare negligible.

As previously noted, the delay vector (z) corresponding to therequirements set is given byz=Fx  (33)while the measurement vector (y) corresponding to the measurements setis given byy=Hx  (34)whereH=[F ₁ ^(T) F ₂ ^(T) . . . F _(Q) ^(T)]^(T)  (35)Because {F₁, F₂, . . . , F_(Q)} is the maximal set of independent rowsof F, every row vector Fk can be expressed as a linear combination ofF₁, F₂, . . . F_(Q).As a resultz=Fx=AHx  (36a)whereA=[Iα ^(T)]^(T)  (36b)and the row vectors α_(k), k=Q+1, Q+2, . . . P are used to express F_(k)in terms of F₁, F₂, . . . , F_(Q) (as discussed above in equation (17)).A General Procedure for Employing a Measurements Set in DeterminingDelay for a Requirements Set

A procedure to deterministically compute the round-trip delay betweenpairs of routers, given by Φ, from measured round-trip delay between theQ number of pairs of routers, identified by Ω, can be defined using thepreceding results. As can be seen, the computed round-trip delay, Δ,between the P pairs of routers in Φ is given byΔ=Ay  (37)where y_(k), k=1, . . . , Q, are delay measurements between the Q pairsof routers in Ω, and Δ_(j), j=1, 2, . . . , P, are delay between the Ppairs of routers in Φ.

FIG. 3 illustrates a process of deterministically computing theround-trip delay between pairs of routers. The process begins with theidentification of the requirement set (step 300). The F matrix is thendetermined from the given requirements set (step 305). Once the F matrixhas been derived, the rank of F is computed (step 310). The rank of Findicates the number of independent rows contained in F. The rank of Fmay be represented, for example, by Q. Once the rank of F has beencomputed, a maximal set of independent rows of F is identified (step315). If the first Q rows of F are not independent (step 320), the rowsof F (as well as router pairs in the requirement set) are reordered suchthat the first subset of Q row (i.e., F₁, F₂, . . . , F_(Q)) areindependent (step 325).

Once the rows of F (and router pairs in the requirements set) have beenreordered, or if such reordering is unnecessary (i.e., the first Q rowsof F are independent) (step 320), the first Q pairs of routers in Φ(router pairs in the requirements set) are copied to form the subset ofΩ, also referred to herein as the measurements set (step 330). Once themeasurement set Ω (or equivalently, the set of independent vectors F1,F2, . . . , FQ) have been identified, row vectors of α_(k), are derivedsuch that F_(k) can be expressed as F_(k)=α_(k1)F₁+α_(k2)F₂+ . . .+α_(kQ)F_(Q). This is done for each value of k=Q+1, Q+2, . . . , P (step335). From these vectors, the A matrix is constructed (step 340). Anetwork monitoring tool is then used to measure round-trip delay betweenpairs of routers identified by Ω, and the measurements are organized inthe form of a column vector, y, of size Q (step 345). The round-tripdelay can then be computed using the vectors thus derived (step 350).The above procedure may be summarized as follows:

-   BEGIN    -   Input requirements set Φ;    -   Determine F matrix from the given requirements set, Φ, and the        routing table;    -   Q=Rank(F);    -   Identify a maximal set of independent rows of F;    -   If necessary, reorder the rows of F (as well as pairs of routers        in Φ), such that the first subset of Q rows, F₁ F₂ . . . F_(Q),        are independent;    -   Copy the first Q number of pairs of routers in Φ to form the        subset Ω;    -   Determine vector α_(k), k=Q+1, . . . , P, such that F_(k) can be        expressed in terms of F₁, F₂, . . . , F_(Q);    -   Construct the A matrix;    -   Measure round-trip delay between router pairs in Ω, and organize        them as y.    -   Compute round-trip delay using the equation Δ=Ay.-   END    It will be noted that a negligible measurement error is preferable.    It will also be noted that only a subset of the non-independent rows    of F (i.e., F_(k), where k is one (or more) of Q+1, . . . , P) may    be of interest, in which case only a subset of the independent rows    of F may be needed to describe such F_(k).

Each of the blocks of the flow diagram of FIG. 3, and those depicted insubsequent figures, may be executed by a module (e.g., a softwaremodule) or a portion of a module or a computer system user. The methodsdescribed herein, the operations thereof and modules for performing suchmethods may therefore be executed on a computer system configured toexecute the operations of the method and/or may be executed fromcomputer-readable media. The method may be embodied in amachine-readable and/or computer-readable medium for configuring acomputer system to execute the method. The software modules may bestored within and/or transmitted to a computer system memory toconfigure the computer system to perform the functions of the module.Alternatively, such actions may be embodied in the structure ofcircuitry that implements such functionality, such as the micro-code ofa complex instruction set computer (CISC), firmware programmed intoprogrammable or erasable/programmable devices, the configuration of afield-programmable gate array (FPGA), the design of a gate array orfull-custom application-specific integrated circuit (ASIC), or the like.

Those skilled in the art will also recognize that the boundaries betweenmodules and operations depicted herein are merely illustrative andalternative embodiments may merge such modules or operations, or imposean alternative decomposition of functionality thereon. For example, theactions discussed herein may be decomposed into sub-operations to beexecuted as multiple computer processes. Moreover, alternativeembodiments may combine multiple instances of a particular operation orsub-operation. Furthermore, those skilled in the art will recognize thatthe operations described in exemplary embodiment are for illustrationonly. Operations may be combined or the functionality of the operationsmay be distributed in additional operations in accordance with theinvention. As will also be apparent to those of skill in the art,methods for determining delay and jitter described herein may employother techniques (similar in effect to those described herein) to makesuch determinations, and such alternative techniques are intended to becomprehended by the methods and apparati discussed herein.

As an example of the operation of such a process, the network of FIG. 2can again be considered, where N=6, M=9. The routes between pairs ofrouters, R_(i) and R_(j), are given in Table 1. Let Φ={(R₃, R₄) (R₃, R₅)(R₄, R₆) (R₅, R₆)}. As beforez=Fx  (38)wherez=[z ₁ z ₂ z ₃ z ₄]^(T)  (39)x=[d ₁ , d ₂ , . . . , d ₉ , s ₁ , s ₂ , . . . , s ₆]^(T)  (40)and the F matrix is a 4×15 matrix of the form:

D₁ D₂ D₃ D₄ D₅ D₆ D₇ D₈ D₉ S₁ S₂ S₃ S₄ S₅ S₆ z₁ 1 1 1 1 z₂ 1 1 1 z₃ 1 11 1 z₄ 1 1 1Because Rank(F)=3, and z₂, z₃ and z₄ are independent row vectors, therow vectors, F_(k), can be re-ordered as those corresponding to z₂, z₃,z₄, z₁. The re-ordered F matrix appears as:

D₁ D₂ D₃ D₄ D₅ D₆ D₇ D₈ D₉ S₁ S₂ S₃ S₄ S₅ S₆ z₂ 1 1 1 z₃ 1 1 1 1 z₄ 1 11 z₁ 1 1 1 1The requirements set Φ is also reordered as {(R₃, R₅) (R₄, R₆) (R₅,R₆)(R₃, R₄)}. Further, Ω={(R₃, R₅), (R_(4 R) ₆), (R₅, R₆)}. Theresulting H matrix appears as:

D₁ D₂ D₃ D₄ D₅ D₆ D₇ D₈ D₉ S₁ S₂ S₃ S₄ S₅ S₆ y₂ 1 1 1 y₃ 1 1 1 1 y₄ 1 11Now, because z₁=z₂+z₃−z₄, the matrix α may be computed:

F₂ F₃ F₄ F₁ 1 1 −1As a consequence, the A matrix appears as:

F₂ F₃ F₄ F₂ 1 F₃ 1 F₄ 1 F₁ 1 1 −1

The computed estimate of delay z=[z₂ z₃ z₄ z₁]^(T) is given by [Δ₂ Δ₃ Δ₄Δ₁]^(T)=Ay=A [y₂ y₃ y₄]^(T).

A Formalized Description of a Method for Calculating Delay and Jitterwithin a Requirements Set Using a Measurements Set

A procedure according to a method of the present invention that computesthe round-trip delay and delay-jitter between every pair of routers inthe requirements set (using every router pair π_(k)=(R_(i), R_(j))εΦ)using measurements between pairs of routers identified by themeasurements set, Ω, is now described using a formal syntax. The schemeis particularly useful when the measurement errors are small and no apriori delay measurements are available.

BEGIN /* The following procedure assumes that the topology and routingtable, */ /* for a given DS codepoint, is known. Where necessary, thisprocedure */ /* is executed for each DS codepoint under consideration.*/ /* If the network topology and/or the routing changes, this procedure*/ /* should be executed afresh with the new topology or new routingtable. */ /* Specify the collection of pairs of routers */ Φ = {π_(k) =(R_(i), R_(j)), k = 1, 2, . . . , P}; /* Construct the delay-componentsvector */ x = [d₁, d₂, . . . , d_(M), s₁, s₂, . . . , s_(N)]^(T); Foreach k = 1, 2, . . . , P, do { Identify the pair of routers π_(k) =(R_(i), R_(j)), π_(k) ∈ Φ; Obtain the route link_path(R_(i), R_(j)) =<L_(p), L_(q), . . . , L_(r)>; Determine the row vector, F_(k), suchthat z_(k) = Delay(R_(i), R_(j)) = F_(k) x }; Construct the P × (N+M)matrix, F; Identify a maximal set of independent rows of F; Q = thenumber of maximal set of independent rows of F; If necessary, reorderrows of F and pairs of routers in Φ such that the first Q rows, F₁, F₂,. . . , F_(Q), are independent; For each k = Q+1, Q+2, . . . , P, do {Determine α_(k,1), α_(k,2) , . . . , α_(k,Q), such that F_(k) =Σ_(i=1,...,Q)(α_(k,i) F_(i)); Construct the row vector α_(k) = [α_(k,1),α_(k,2), . . . , α_(k,Q)] }; Copy the first Q pairs of routers from Φ toform the subset Ω; Identify the start time t₀, and the measurementinterval, dt; t₀ = start_time; /* start_time will most likely be 0(start at time 0) */ While (!finished) { For each k = 1, 2, . . . , Q,do { During the time interval, (t₀ = t₀+dt), use a network managementtool (e.g., ANMT) to measure average round-trip delay, y_(k), anddelay-jitter, γ_(k), between the pair of routers, π_(k) = (R_(i), R_(j))}; /* Estimate round-trip delay vector, Δ_(k), for each pair of routersin Φ */ For each router pair in Φ { For each k = 1, 2, . . . , Q, do {Δ_(k) = y_(k) }; For each k = Q+1, Q+2, . . . , P, do { Δ_(k) =Σ_(i=1,...,Q)(α_(k,i) y_(i)) }; } /* Compute round-trip delay-jitter,σ_(k), for each pair of routers in Φ */ For each router pair in Φ { Foreach k = 1, 2, . . . , Q, do { σ_(k) = γ_(k) }; For each k = Q+1, Q+2, .. . , P, do { σ_(k) = √ Σ_(i=1,...,Q)(α_(k,i) ^(a)γ_(i) ²) }; } t₀ =t₀+dt } ENDAs noted, the above scheme is applicable where no other delaymeasurements or a priori estimates are available and assumes that themeasurement error is negligible.

As an example, network 200 may again be considered (network 200 has N=6,M=9, as noted). The routes between all pairs of routers, R_(i) andR_(j), are given in Table 1. Using the technique just described, thenetwork can be analyzed as follows. Let Φ={π₁=(R₃, R₄), π₂=(R₃, R₅),π₃=(R₄ R₆), π₄=(R₅, R₆)}. The delay-component vector, x=[d₁, d₂, . . . ,d₉, s₁, s₂, . . . , s₆]^(T). Becausez ₁=Delay(R ₃ , R ₄)=s ₃ +d ₆ +d ₈ +s ₄  (41a)z ₂=Delay(R ₃ , R ₅)=s ₃ +d ₆ +s ₅  (41b)z ₃=Delay(R ₄ , R ₆)=s ₄ +d ₈ +d ₉ +s ₆  (41c)z ₄=Delay(R ₅ , R ₆)=s ₅ +d ₉ +s ₆  (41d)the delay vector, z=[z₁ z₂ z₃ z₄]^(T), may be written as z=Fx, where the4×15 matrix F is:

D₁ D₂ D₃ D₄ D₅ D₆ D₇ D₈ D₉ S₁ S₂ S₃ S₄ S₅ S₆ z₁ 1 1 1 1 z₂ 1 1 1 z₃ 1 11 1 z₄ 1 1 1Because z₂, z₃ and z₄ are independent row vectors, Q=3. The row vectors,F_(k), are re-ordered as those corresponding to z₂, z₃, z₄, z₁. There-ordered F is:

D₁ D₂ D₃ D₄ D₅ D₆ D₇ D₈ D₉ S₁ S₂ S₃ S₄ S₅ S₆ z₂ 1 1 1 z₃ 1 1 1 1 z₄ 1 11 z₁ 1 1 1 1Because z₁=z₂+z₃−z₄, the vector α₁=[1 1 −1]. As a result, Ω={π₂=(R₃,R₅), π₃=(R₄ R₆), π₄=(R₅, R₆)}, with the corresponding H matrix of theform given in the previous example. Next, t₀ is set to 0, and dt is setto 60 (seconds), for example.

During the interval (t₀, t₀+dt), a network probe tool is used to measureround-trip delay and jitter between pairs of routers, {π₂=(R₃, R₅),π₃=(R₄ R₆), π₄=(R₅, R₆)} to yield delay measurements, y₂, y₃, and y₄,and delay-jitter measurements, γ₂, γ₃, and γ₄. An estimate of delaybetween pairs of routers in Φ is given byΔ₂ =y ₂  (42a)Δ₃ =y ₃  (42b)Δ₄ =y ₄  (42c)Δ₁ =y ₂ +y ₃ −y ₄  (42d)An estimate of delay-jitter between pairs of routers in Φ is given byσ₂=γ₂  (43a)σ₃=γ₃  (43b)σ₄=γ₄  (43c)σ₁=√(γ₂ ²+γ₃ ²+γ₄ ²)  (43d)Measurement and estimation can then be repeated with t₀=t₀+dt.II. Measurements for a Large Number of Router Pairs in a Network withKnown Topology

A method for estimating delay, including packet transfer delays and ANMTprocessing delays, for a requirements set containing a large number ofnodes according to certain embodiments of the present invention is nowdescribed. Such a method accounts for the delay encountered intransferring information over links in a network, and also accounts fordelays within the end-point nodes, and intermediate nodes along the pathbetween the end-point nodes. Estimation of the vector x, includesIP-level transfer delay, d_(j), over each link, L_(j), and of processingdelay caused by the network monitoring tool(s) employed within eachrouter R_(i) (defined as s_(i)) may be performed used a method describedbelow. The need to estimate the IP-level delay over each link stems fromthe fact that a measurement between the pair of routers corresponding toa link, made using a network monitoring tool, generally include the timerequired to generate, receive and process probe packets within the pairof source and destination routers. To that end, an estimate of delayincurred due to protocol processing (referred to generically herein asIP-level transfer delay) over a link better reflects the delay that atypical packet (referred to generically herein as an IP packet)encounters over each hop.

If an estimate of delay due to processing within each router isadditionally available, then one may estimate the delay experienced byprobe packets between any pair of routers. In other words, once anestimate of vector x is available, an estimate of Delay(R_(p), R_(q))may be obtained, because Delay(R_(p), R_(q))=F_(k)x for any specifiedpair of routers, (R_(p), R_(q)).

The Measurements Set, Ω

To review, the measurements set, Ω, corresponding to all pairs ofrouters may be defined as the minimal subset of pairs of routers,between which delay should be measured so that delay between every pairof routers in the network can be obtained. The number ofrouter-to-router delay measurements required to estimate delay betweenan arbitrarily specified subset of pairs of routers is at most M+N. Asnoted:Q=|Ω|=Rank(F)<min(|Φ|, N+M)  (45)This is so, because the matrix, F, is a |Φ|×(M+N) matrix, the rank ofwhich, with respect to Q, is at most the lesser of |Φ| or (M+N).

The approach outlined previously may be used to obtain the subset ofpairs of routers, Ω, between which delay and jitter measurements aremade, corresponding to the set of all pairs of routers,Φ={(R _(i) , R _(j)), i=1, 2, . . . N, j=1, 2, . . . , N, i<j}  (46)

Such an approach requires that Delay(R_(i), R_(j)) be expressed asF_(k)x, for each pair of routers, k=1, 2, . . . , N*(N−1)/2, and themaximal subset of linearly independent row vectors, F_(k) be identifiedby determining the maximal set of independent set of row vectors. As canbe seen, this can be a computationally complex procedure because theprocedure involves vectors of order N²(O(N²)).

Alternatively, one may directly identify the subset Ω. A methodaccording to one embodiment of the present invention that identifies asmallest subset of pairs of routers, such that the delay between everyother pair of routers can be expressed as a linear combination of delaybetween the pairs of routers in subset, Ω, is described below. The sizeof the matrix, Q, is no more than N+M. Further, the corresponding H=[H₁^(T) H₂ ^(T) . . . H_(Q) ^(T)]^(T), is of full rank, Q.

It will be noted that the size of Ω (i.e., Q) can be in fact smallerthan N+M, if for some router, R_(i), there does not exist a path betweentwo routers, R_(p) and R_(q), such that the route between R_(p), andR_(q), passes through R_(i). That is, for some R_(i),router_path(R _(p) , R _(q))≠<R _(p) , . . . , R _(i) , . . . R _(q)>for all R _(p) ≠R _(i) and R _(q) ≠R _(i)  (47)The immediate consequence of this observation is that it would not bepossible to isolate, and thus separately estimate, the delay withinrouter R_(i) due to processing required by the network monitoringsoftware, except in combination with delays over links connecting R_(i)to its neighboring routers. The details of such a technique are nowdiscussed.A Measurement Scheme for Estimating Delay Components

A scheme according to an embodiment of the present invention capable ofprocessing a requirements set having a large number of nodes is nowdescribed. The scheme provides an estimate of the delay and jitterbetween any arbitrarily specified pair of routers based on measurementsbetween a subset of pairs of routers.

FIG. 4 is a flowchart depicting such a scheme, which includes the threesteps illustrated therein.

-   -   1. Identification of a minimal subset of router-to-router        measurements, Ω, that must be made in order to obtain an        estimate of delay components, x=[d₁, d₂, . . . , d_(M), s₁, s₂,        . . . , s_(N)]^(T) (step 400).    -   2. The use of a standard technique to estimate delay components        (step 410).    -   3. The estimation of the delay between the given pair of routers        using an equation that expresses router-to-router delay in terms        of the delay components (step 420).        The Identification of Router-to-Router Measurements

Letting Ω be the minimal subset of Q number of pairs of routers betweenwhich measurements are made, the Q measurement,y_(k)=z_(k)+v_(k)=H_(k)x+v_(k), k=1, 2, . . . , Q, can be re-written asy=Hx+v  (48)wherey=[y ₁ , y ₂ , . . . , y _(Q)]^(T)  (49a)v=[v ₁ , v ₂, . . . , v_(Q)]^(T)  (49b)H=[H ₁ ^(T) H ₂ ^(T) . . . H _(Q) ^(T)]^(T)  (49c)Thus, a method according to one embodiment of the present inventionidentifies the subset Ω such that the resulting matrix H has the maximumpossible rank, Q=|Ω|. In such an embodiment, row vectors are H₁, H₂, . .. , H_(Q) are independent. This can be represented, for example, in thefollowing manner.

FIG. 5A is a flow diagram illustrating a method according to oneembodiment of the present invention that identifies the subset Ω suchthat the resulting matrix H has the maximum possible rank. The processbegins by initializing the set Ω to empty. The set Ω will contain routerpairs between which performance is to be measured (step 500). Next, aset Θ is initialized to empty (step 502). This set will contain routerpairs for which processing time cannot be separately estimated. Thefirst M measurements are between all adjacent routers, one for eachlink, and proceeds as follows. For each link, L_(k), k=1, 2, . . . , M(step 504), the pair of routers (R_(i), R_(j)) that are the “end-points”of that link are identified (step 506) and added to the set Ω (step508). Then, for each router (1, . . . , N), say R_(k) (step 510), thecollection of adjacent router are examined to determine where there is apair of routers adjacent to R_(k) such that a path between them passesthrough R_(k) (step 512). If such a pair is successfully identified(step 514), then this pair is added to the set Ω (step 516). Otherwise,it may be concluded that it is not possible to separately assess delaydue to processing within the router R_(k). In this case, R_(k) is addedto the set Θ (step 518).

Once the sets Ω and Θ are thus determined, all the router pairs in Ω areexamined. For each router pair (step 520), row H_(k) of the H matrix isdetermined, such that Z_(k)=Delay (R_(i), R_(j))=H_(k)x (step 522). Oncethe elements of the H matrix have been determined, the H matrix isformed from the elements so determined (step 524). Once the H matrix hasbeen formed, the rank Q of the H matrix is determined (step 526). It canbe verified that Q must be N+M−|Ω|.

The above method for identifying the subset Ω such that the resultingmatrix H has the maximum possible rank is now outlined using a formalrepresentation.

BEGIN /* proposed set of pairs of routers, between which performance isto be measured */ Ω = { }; /* set of routers, ANMT processing time forwhich cannot be separately estimated */ Θ = { }; /* first M measurementsare between adjacent routers */ For all links L_(k), k = 1, 2, . . . ,M, do { (R_(i), R_(j)) = identify_router_pair(L_(k)) Ω = Ω + {( R_(i),R_(j))}; }; /* N − |Θ| measurements are between certain pairs of routersadjacent to an R_(i) */ For all routers R_(k), k = 1, 2, . . . , N, do {pair_identified = false; φ(R_(k)) = set_of_all_adjacent_routers(R_(k));For all (R_(i) ∈ φ(R_(k))) and (R_(j) ∈ φ(R_(k))) and (R_(i) ≠ R_(j)),do { If (router_path(R_(i), R_(j)) = <R_(i), R_(k), R_(j)>) then { Ω =Ω+ {(R_(i), R_(j))}; pair_identified = true; exit /* from the currentloop */ }; }; if pair_identified = false then { Θ = Θ + {R_(k)} } }; k =1; For all pairs of routers (R_(i), R_(j)) ∈ Ω do { determine H_(k) suchthat Delay(R_(i), R_(j)) = H_(k) x k = k + 1 } Form matrix H; /* Rank(H)should be N+M−|Θ| */ Q = Rank(H) END

Before establishing the adequacy of the above method, it is noted thatusing a network performance measurement tool, such as ANMT, allows probepackets to be sent through a given router using “loose source routing”.In such cases, it is possible to ensure that the subset Θ is empty.

Proof that the (N+M|Θ|)×(N+M) matrix, H, resulting from an applicationof the above method has full rank (i.e., N+M−|Θ|) is now given.

If Θ is non-empty, the columns of H can be re-arranged by moving columnscorresponding to the variables s_(i) for all i, such that R_(i)εΘ, tothe end of matrix H. The resulting H matrix then has the followingstructure:

H = [ I H₁₂ H₁₃ ] (50) [ H₂₁ H₂₂ H₂₃ ]where I is an M×M identity matrix, and H₁₂, H₁₃, H₂₁, H₂₂ and H₂₃ aresub-matrices of appropriate dimension (M×N−|Θ|, M×|Θ|, N−|Θ|×M,N−|Θ|×N−|Θ|, and N−|Θ|×|Θ|, respectively). In particular, sub-matricesH₁₃ and H₂₃ correspond to variables, s_(i), such that R_(i)εΘ. If Θ isempty, then H₁₃ and H₂₃ are not present.

The individual rows of the sub-matrix[H ₂₁ H ₂₂ H ₂₃]  (51)are now considered. Each row h_(M+k) represents a measurement of thetypeDelay(R _(p) , R _(q))=s _(p) +d _(i) +d _(j) +s _(q)  (52)where R_(p) and R_(q) are routers that are directly connected to routerR_(k) using links L_(i) and L_(j). Consider rows h_(i) and h_(j), whichcorrespond to measurements:Delay(R _(p) , R _(k))=s _(p) +d _(i) +s _(k)  (53)Delay(R _(k) , R _(q))=s _(k) +d _(j) +s _(q)  (54)respectively. Thus, setting row h_(M+k)=(h_(i)+h_(j)−h_(M+k)) yieldsh_(M+k)=[0 0 . . . 0 2 0 . . . 0], where the “2” is in the M+k^(th)position.

Repeating this operation for every row M+k=M+1, M+2, . . . , M+N−|Θ|results in a matrix H′ of the form:

H′ = [ I H₁₂ H₁₃ ] (55) [ 0 2*I 0 ]As is thus made evident, the matrix H′ has full rank (i.e.,Rank(H′)=M+N−|Θ|). Because Rank(H′)≦Rank(H)≦M+N−|Θ|, it follows thatRank(H)=M+N−|Θ|.

Delay between any pair of routers, Delay(R_(p), R_(q)), can be expressedas a linear combination of the collection of {Delay(R_(i), R_(j)),(R_(i), R_(j))εΩ}. Equivalently, it can be shown that if Delay(R_(p),R_(q))=F_(k)x, for some F_(k), then F_(k)=α_(k)H, where the M+N−|Θ|×M+Nmatrix corresponds to the subset Ω. From Equation 52, matrix H isdefined as

H = [ I H₁₂ H₁₃ ] (56) [ H₂₁ H₂₂ H₂₃ ]Letting

C = [ I 0 ] (57a) [ H₂₁ −I ]it may be shown that

H′ = CH = [ I H₁₂ H₁₃ ] (57b) [ 0 2*I 0 ]This is so because H₂₁ H₁₂−H₂₂=2*I, and H₂₁ H₁₃−H₂₃=0. Defining:

B = [ I −1/2 H₁₂ ] (58a) [ 0 1/2 I ]H″ can be derived as follows:

H″ = BH″ = BCH = [ I 0 H₁₃ ] (58b) [ 0 I 0 ]As noted, the columns of the sub-matrix, H₁₃, correspond to variables,s_(k), for each router, R_(k), where R_(k)εΘ. Such a router is onethrough which no route passes, except as a source or destination router.

The delay between any given pair of routers, R_(p) and R_(q), can thusbe expressed in terms of delay components, assuming thatlink_path(R_(p), R_(q))=<L_(u), L_(v), . . . , L_(w)> is given by:$\begin{matrix}{{{Delay}\quad\left( {R_{p},R_{q}} \right)} = {s_{p} + d_{u} + d_{v} + \ldots + d_{w} + s_{q}}} & {\quad\text{(59a)}} \\{= {F_{x}\quad x}} & {\quad\text{(59b)}}\end{matrix}$whereF _(k)=[0 . . . 0 1 0 . . . 0 1 0 . . . 0 1 0 . . . 0 1 0 . . . 0 1 0 .. . 0]  (59c)and the 1's in F_(k) appear in columns u, v, . . . , w, and M+p and M+q.As is now shownF _(k) x=φ _(k) H″x=φ _(k) B C Hx,  (60)where φ_(k) is simply the first M+N−|Θ| components of the row vectorF_(k). To prove that F_(k)x=φ_(k) H″ x=φ_(k) B C H x, the followingcases need to be considered:

-   -   1. Case 1: If |Θ|=0, then, there are no columns corresponding to        sub-matrix H₁₃. As a result, H″=B C H=I and φ_(k)=F_(k)    -   2. Case 2: If |Θ|>0, then if R_(p)εΘ, the first two terms in the        expansion of Delay(R_(p), R_(q)) can be combined to yield as        Delay(R_(p), R_(q))=(s_(p)+d_(u))+d_(v)+ . . . +d_(w)+s_(q). The        u^(th) row of H″x is exactly (s_(p)+d_(u)). Similarly, if        R_(q)εΘ, the last two terms in its expansion can be combined to        yield Delay(R_(p), R_(q))=s_(p)+d_(u)+d_(v)+ . . .        +(d_(w)+s_(q)). The w^(th) row of H″x is exactly (s_(q)+d_(w)).        As a result, entries in the last |Θ| columns of F_(k) may be        dropped.        As a result, from (60) above, F_(k)x=α_(k)Hx, where        α_(k)=φ_(k) B C  (61)        Thus, delay between any pair of routers can be expressed in        terms of measured delay between pairs of routers, identified by        Ω. An example employing the preceding results is now given.

The example network given in FIG. 2 is now considered. Note once againthat N=6, M=9. The routing table is given in Table 1. Using the methoddescribed previously, the collection of pairs of routers is identified:$\quad\begin{matrix}\begin{matrix}{\Omega = \left\{ {\left( {R_{1},R_{2}} \right),\left( {R_{2},R_{3}} \right),\left( {R_{1},R_{4}} \right),\left( {R_{2},R_{4}} \right),\left( {R_{2},R_{5}} \right),} \right.} \\{\left( {R_{3},R_{5}} \right),\left( {R_{3},R_{6}} \right),\left( {R_{4},R_{5}} \right),\left( {R_{5},R_{6}} \right),\left( {R_{1},R_{3}} \right),} \\\left. {\left( {R_{2},R_{6}} \right),\left( {R_{1},R_{5}} \right),\left( {R_{4},R_{6}} \right)} \right\}\end{matrix} & (62)\end{matrix}$Further, since no path passes through routers R₁ or through R₆,Θ={R ₁ , R ₆}  (63)Because |Θ|=2, the 13×15 matrix H is given by:

D₁ D₂ D₃ D₄ D₅ D₆ D₇ D₈ D₉ S₁ S₂ S₃ S₄ S₅ S₆ Y₁  1 1 1 Y₂  1 1 1 Y₃  1 11 Y₄  1 1 1 Y₅  1 1 1 Y₆  1 1 1 Y₇  1 1 1 Y₈  1 1 1 Y₉  1 1 1 Y₁₀ 1 1 11 Y₁₁ 1 1 1 1 Y₁₂ 1 1 1 1 Y₁₃ 1 1 1 1Because Θ={R₁, R₆}, the columns are re-arranged such that the columnscorresponding to s₁, and s₆ are listed last. This results in a new Hmatrix corresponding to a new definition of the vector of delaycomponents, x=[d₁, d₂, . . . , d₉, s₂, . . . , s₅, s₁, s₆]^(T):

D₁ D₂ D₃ D₄ D₃ D₆ D₇ D₈ D₉ S₂ S₃ S₄ S₅ S₁ S₆ If C = Y₁  1 1 1 1 Y₂  1 11 1 Y₃  1 1 1 1 Y₄  1 1 1 1 Y₅  1 1 1 1 Y₆  1 1 1 1 Y₇  1 1 1 1 Y₈  1 11 1 Y₉  1 1 1 1 Y₁₀ 1 1 1 1 1 1 −1 Y₁₁ 1 1 1 1 1 1 −1 Y₁₂ 1 1 1 1 1 1 −1Y₁₃ 1 1 1 1 1 1 −1 and B = D₁ D₂ D₃ D₄ D₅ D₆ D₇ D₈ D₉ S₂ S₃ S₄ S₅ S₁ S₆Y₁  1 −1/2 1 1 Y₂  1 −1/2 −1/2 1 Y₃  1 −1/2 1 1 Y₄  1 −1/2 −1/2 1 Y₅  1−1/2 −1/2 1 Y₆  1 −1/2 −1/2 1 Y₇  1 −1/2 1 1 Y₈  1 −1/2 −1/2 1 Y₉  1−1/2 1 1 Y₁₀   1/2 1 Y₁₁   1/2 1 Y₁₂   1/2 1 Y₁₃   1/2 1Thus, it can be seen that matrix H″ has full rank (i.e.,N+M−|Θ|=6+9−2=13), and that Rank(H)=13.

The fact that |Θ|=2, or that the rank of matrix H is 13, and not 15,implies that one is not able to estimate all N+M=15 parametersindividually. In light of the present invention, a close examination ofthe network reveals that there is no way that one can estimate d₁, d₃,and s₁, independently (or d₇, d₉, and s₆). In every measurement thatinvolves d₁, parameters s₁ and d₁ appear together (and similarly, d₃ &s₁, d₇ & s₆, d₉ & s₆). This is so because there is no path in thenetwork that passes through node R₁ (or through R₆).

A method according to one embodiment of the present invention, however,does provide a measure of d₁+s₁, d₃+s₁, d₇+s₆, d₉+s₆. This is not aconstraint because delay between any pair of routers can be expressed interms of measured delay between pairs of routers in Ω. In particular,$\quad\begin{matrix}{{{Delay}\quad\left( {R_{1},R_{6}} \right)} = {s_{1} + d_{3} + d_{8} + d_{9} + s_{6}}} & \left( {64a} \right) \\{= {F_{({1,6})}x}} & \left( {64b} \right) \\{= {\left\lbrack \quad\begin{matrix}0 & 0 & 1 & 0 & 0 & 0 & 0 & 1 & 1 & 0 & 0 & 0 & 0 & 1 & 1\end{matrix}\quad \right\rbrack x}} & \left( {64c} \right) \\{= {\left\lbrack \quad\begin{matrix}0 & 0 & 1 & 0 & 0 & 0 & 0 & 1 & 1 & 0 & 0 & 0 & 0\end{matrix}\quad \right\rbrack\quad H^{''}x}} & \left( {64d} \right) \\{= {\phi_{({1,6})}{BCHx}}} & \left( {64e} \right) \\{= {\alpha_{({1,6})}{Hx}}} & \left( {64f} \right) \\{{{Delay}\quad\left( {R_{1},R_{6}} \right)} = {s_{3} + d_{6} + d_{8} + s_{4}}} & \left( {65a} \right) \\{= {F_{({3,4})}x}} & \left( {65b} \right) \\{= {\left\lbrack \quad\begin{matrix}0 & 0 & 0 & 0 & 0 & 1 & 0 & 1 & 0 & 0 & 1 & 1 & 0 & 0 & 0\end{matrix}\quad \right\rbrack x}} & \left( {65c} \right) \\{= {\left\lbrack \quad\begin{matrix}0 & 0 & 0 & 0 & 0 & 1 & 0 & 1 & 0 & 0 & 1 & 1 & 0\end{matrix}\quad \right\rbrack\quad H^{''}x}} & \left( {65d} \right) \\{= {\phi_{({3,4})}{BCHx}}} & \left( {65e} \right) \\{= {\alpha_{({3,4})}{Hx}}} & \left( {65f} \right)\end{matrix}$where x=[d₁, d₂, . . . , d₉, s₂, . . . , s₅, s₁, s₆]^(T), and H, C, andB are given by their respective definitions immediately preceding. Thus,α_((1,6))=[0 0 1 0 0 0 0 1 1 0 0 0 0] B C=[0 0 0 0 0 0 0 −1 0 0 0 11]  (66a)α_((3,4))=[0 0 0 0 0 1 0 1 0 0 1 1 0] B C=[0 0 0 0 0 1 0 0 −1 0 0 01]  (66b)In other words, Delay(R₁, R₆)=−z₈+z₁₂+z₁₃ and Delay(R₁, R₆)=z₆−z₉+z₁₃,where z_(k)=H_(k)x corresponds to the k^(th) pair of router in Ω. As aresult, the delay between any given pair of routers can be obtained fromthe set of orthogonal delay components.A Formalized Description of a Method for Calculating Delay and JitterBetween an Arbitrarily Specified Pair of Routers

A procedure according to a method of the present invention that computesround-trip delay and delay-jitter between any arbitrarily specifiedcollection of router pairs using measurements between router pairs inthe measurements set is now described. The theoretical underpinnings ofthe method used to construct the measurements set and the method ofcalculating delay and jitter from the measurements are discussed above,in the description immediately preceding. Such a procedure isparticularly useful when the number of pairs of routers in therequirements set is large. Such a procedure is applicable whenmeasurement errors are negligible, and a-priori measurements are notavailable.

FIG. 5B is a flow diagram illustrating such a method. It is assumed thatthe network topology and routing table, for a given DS codepoint, isknown. Where necessary, such a procedure is executed for each DScodepoint under consideration. If the network topology and/or therouting table changes, the procedure described presently should beexecuted afresh with the new topology or with new routing table. Theprocess begins with the specification of the collection of router pairs(step 540) and the definition of the delay-components vector (step 542).Next, the sets Ω and Θ are initialized to empty (steps 544 and 546,respectively). The set Ω will contain router pairs between whichperformance is to be measured, while the set Θ will contain router pairsfor which processing time cannot be separately estimated.

The first M measurements are between adjacent routers, one for eachlink, and proceeds as follows. For each link, L_(k), k=1, 2, . . . , M(step 548), the pair of routers (R_(i), R_(j)) that are the “end-points”of that link are identified and added to the set Ω (step 550). The nextset of up to N measurements are then identified. These measurements arebetween a pair of routers (R_(i), R_(j)), each of which is adjacent torouter R_(k). Such a pair is included in the measurements set providedthere exists at least one path from R_(i) to R_(j), which passes throughR_(k). As a result, this may result in N−|Θ| measurements only. For eachrouter R_(k), k=(1, . . . , N) (step 552), the collection of adjacentrouters (φ(R_(k))) are examined to determine where there is a pair ofrouters adjacent to R_(k) such that a path between them passes throughR_(k) (step 554). For each router pair in (φ(R_(k)) (step 556), thepreceding determination is made. If such a pair is successfullyidentified (step 558), then this pair is added to the set Ω (step 560).Otherwise, if no such pair is successfully identified (step 561), it maybe concluded that it is not possible to separately assess delay due toprocessing within the router R_(k), and so R_(k) is added to the set Θ(step 562). Construction of the measurements set, Ω, is thus completed.The size of Ω (represented, e.g., by Q) is given by N+M−|Θ| (step 564).

Once the sets Ω and Θ are thus determined, the router pairs in Ω areexamined. For each router pair (step 566), a unique number associatedwith that router pair and row H_(k) of the H matrix is determined, suchthat Z_(k)=Delay (R_(i), R_(j))=H_(k)x (step 568). Once the elements ofthe H matrix have been determined, the H matrix is formed from theresulting elements and the rank of H determined (step 570). It can beverified that Q must be N+M−|Ω|. If such is not the case (step 572), theprocedure is exited.

Next, the columns of H are re-arranged by moving the columnscorresponding to s_(i) (for all i, such that R_(i)εΘ) to the end of H(step 574). Also at this time, the components of x are re-arranged bymoving the components corresponding to s_(i) (for all i, such thatR_(i)εΘ) to the end of x. Matrices B and C are then constructed, andtheir matrix product D computed (step 576). For each pair of routersπ_(k)=(R_(p), R_(q)) in the requirements set, Φ (step 578), constructthe row vector, F_(k), and the row vector, φ_(k) (step 580). Next, α_(k)is computed, ensuring that Delay(R_(p), R_(q))=F_(k)x=α_(k)Hx (step582).

Now that initialization is complete, measurements can now be taken, anddelay and jitter between router pairs contained in the requirements setcalculated. While further measurement time intervals remain (i.e.,measurements are to continue being taken) (step 584), and for eachrouter pair π_(k)=(R_(i), R_(j)) in the measurements set, Ω (step 586),measure average round-trip delay (y_(k)) and delay-jitter (γ_(k))between each pair of routers (π_(k)=(R_(i), R_(j))) during the giventime interval (step 588). This can be accomplished, for example, usingan ANMT. Additionally, for each router pair (R_(p), R_(q)) in therequirements set, Φ (step 590), calculate round-trip delay vector(Δ_(k)) and jitter (σ_(k)) (step 592). This is repeated until nomeasurement time intervals remain.

The above method for computing round-trip delay and delay-jitter betweenany arbitrarily specified collection of router pairs using measurementsbetween router pairs in the measurements set is now outlined using aformal representation.

BEGIN /* The following assumes that the network topology and routingtable, */ /* for a given DS codepoint, is known. */ /* Where necessary,such a procedure is executed for each DS codepoint, */ /* underconsideration. /* If the network topology and/or the routing tablechanges, this procedure */ /* is preferably executed afresh with the newtopology or with new */ /* routing table. */ /* Specify the collectionof pairs of routers. */ Φ = {π_(k) = (R_(i), R_(j)), k = 1, 2, . . . ,P}; /* Define the delay-components vector. */ x = [d₁, d₂, . . . ,d_(M), s₁, s₂, . . . , s_(N)]^(T); /* Initialize the measurements set(consisting of pairs of routers, */ /* between which delay/jitter ismeasured) to an empty set. */ Ω = { }; /* Initialize the set of routers(for which ANMT processing delay */ /* cannot be separately estimated)to an empty set. */ Θ = { }; /* Identify the first M measurements. Theseare one for each link, */ /* and between adjacent routers. It will benoted that the procedure */ /* identify_router_pair(L_(k)) identifiesthe adjacent routers directly */ /* linked using link, L_(k). For link(L_(k)) where k = 1, 2, . . . , M, do { (R_(i), R_(j)) =identify_router_pair(L_(k)); Ω = Ω + {( R_(i), R_(j))} }; /* Identifythe next set of up to N measurements. These measurements */ /* arebetween a pair of routers (R_(i), R_(j)), each of which is adjacent to*/ /* router R_(k). Such a pair is included in the measurements setprovided */ /* there exists at least one path from R_(i) to R_(j) whichpasses through R_(k). */ /* As a result, this may result in N − |Θ|measurements only. It will be */ /* noted that the procedureset_of_all_adjacent_routers(R_(k)) identifies */ /* all routers that areadjacent to the specified router, R_(k). */ For all routers R_(k), k =1, 2, . . . , N, do { pair_identified = false; φ(R_(k)) =set_of_all_adjacent_routers(R_(k)) ; For all R_(i) ∈ φ(R_(k)), R_(j) ∈φ(R_(k)), R_(i) ≠ R_(j), do { If (router_path(R_(i), R_(j)) = <R_(i),R_(k), R_(j)>) { Ω = Ω + {(R_(i), R_(j))}; pair_identified = true; exit}; If (pair_identified = false) { Θ = Θ + {R_(k)} } }; /* Constructionof the measurements set, Ω, is now complete. */ /* The size of the setis N+M−|Θ|. */ Q = |Ω|; /* For each pair of routers in the measurementsset, (R_(i), R_(j)), */ /* associate a unique number, and determine thecorresponding row vector, */ /* H_(k) such that round-trip delay betweenthe router pair (R_(i), R_(j)) */ /* can be expressed as H_(k) x. */ k =1; For all pairs of routers (R_(i), R_(j)) ∈ Ω, { determine H_(k) suchthat Delay(R_(i), R_(j)) = H_(k) x; k = k +1 }; /* Form the matrix, H,and compute its rank. */ /* The Rank(H) should be Q = N+M−|Θ|. */ H =[H₁ ^(T) H₂ ^(T) . . . H_(Q) ^(T)]^(T); If {Rank(H) != Q) { exit /* EXITremaining procedure */ } /* Re-arrange columns of H as follows: movecolumns corresponding to */ /* variables, s_(i), for all i, such thatR_(i) ∈ Θ, to the end of matrix H. */ /* Similarly, re-arrange thecomponents of the vector of */ /* delay components, x. */ H = [ I H₁₂H₁₃ ]; [ H₂₁ H₂₂ H₂₃ ] /* Construct matrices, B and C, and compute thematrix product D = BC. */ C = [ I 0 ]; [ H₂₁ −I ] B = [ I −1/2 H₁₂ ]; [0 1/2 I ] D = B C; /* For each pair of routers π_(k) = (R_(p), R_(q)) inthe requirements set, Φ, */ /* construct the row vector, F_(k) and therow vector, φ_(k). */ Construct F_(k) such that Delay(R_(p), R_(p)) =F_(k) x; From F_(k) construct φ_(k) by retaining the first Q componentsof F_(k); /* Compute α_(k). This ensures that Delay(R_(p), R_(q)) =F_(k) x = α_(k) H x. */ α_(k) = φ_(k) D; /* The initialization steps arenow complete. */ /* Measurements repeatedly and calculate delay andjitter between router */ /* pairs contained in the requirements set. *//* t₀ is the start time and dt is the measurement interval. */ t₀ =start_time; while (!finished) { For each k = 1, 2, . . . , Q, do {During the time interval, (t₀, t₀ +dt), use ANMT to measure averageround-trip delay, y_(k), and delay-jitter, γ_(k), between each pair ofrouters, π_(k) = (R_(i), R_(j)) contained in Ω }; /* For each pair ofrouters, π_(k) = (R_(p), R_(q)), in the requirements set, Φ, */ /*calculate round-trip delay vector, Δ_(k), and jitter, σ_(k) */ For eachk = 1, 2, . . . , |Φ|, do { Δ_(k) = Σ_(i=1,...,Q)(α_(k,i) y_(i)); σ_(k)= √Σ_(i=1,...,Q)(α_(k,i) ² γ_(i) ²) }; /* Repeat the above for the nexttime interval. */ t₀ = t₀+dt } ENDIII. Estimating Delay and Jitter in an Enterprise Network Coupled to aBackbone Having an Unknown Topology

The various embodiments of a method according to the present inventiondescribed previously provide an efficient measurement scheme to estimateround-trip delay and jitter in an enterprise network whose topology androuting is known. These methods may be extended to monitor delay andjitter in an enterprise network that uses a service-provider (SP)backbone network having topology and routing that are not known. Anembodiment of the present invention, such as is now described,illustrates how the preceding method can be used in networks whereenterprise routers are connected through a backbone network of a serviceprovider (SP), the topology or routing of which is unknown (and cannotbe discovered). In this portion of the discussion, the network isreferred to as an enterprise network in order to distinguish theenterprise network from the service provider's backbone. In a similarfashion, elements of the enterprise network are referred to as being apart thereof (e.g., enterprise routers and enterprise links).

More specifically, a method according to one embodiment of the presentinvention discovers the collection of ingress/egress routers of thebackbone or service-provider network used to interconnect the enterpriserouter. Based on the resulting incomplete knowledge of backbone networktopology, a model of the backbone network together with the manner inwhich enterprise traffic is routed through the service-provider networkcan be generated. Such a measurement scheme:

-   -   1. can be used in the current context to estimate delay and        delay-jitter between specified pairs of enterprise routers,    -   2. can be used to estimate various delay components between a        given pair of enterprise routers, and    -   3. can provide an efficient method estimating delay and jitter        between a large number of pairs of enterprise routers.

Thus, a relatively simple method is described that identifies therelevant set of backbone routers and their interconnection to enterpriserouters. Such a method allows the modeling of the relevant portion ofthe backbone network together with its interconnection to enterpriserouters. The resulting model of the network is characterized by N+mrouters and N+m*(m−1)/2 links, where m is the number of backbone routersand N is the number of enterprise routers. It can be shown that, if therequirements set contains P pairs of enterprise routers, then the sizeof a measurements set is no more than min(P, N+m*(m−1)/2). Because m isthe number of backbone routers, the technique is particular efficient ifthe extent of “clustering” is significant, or equivalently, m is small.

A Model of the Enterprise Network

The enterprise network is assumed to include N routers, R_(i), i=1, 2, .. . , N, that are connected to one or more backbone routers using links,L_(j), j=1, 2, . . . , N. It is assumed that the number of links used toconnect an enterprise network to the backbone network is 1, and that theenterprise routers are themselves not connected to each other. Whilethese assumptions are not mandatory (i.e., such topological features canbe accounted for), such are assumed for simplicity of the followingdiscussion. It is also assumed that (a) the identity of backbonerouters, and (b) the interconnection of enterprise routers to thebackbone routers are unknown.

FIG. 6 is a block diagram illustrating an exemplary enterprise network600 in which a method according to an embodiment of the presentinvention can be practiced. Network 600 includes a backbone 610 androuters 620(1)-(K) and 620(K)-(N). Routers 620(1)-(K) and 620(K)-(N) arecoupled to backbone 610 by links 630(1)-(K) and 630(K)-(N),respectively. It will be noted that routers 620(1)-(K) and 620(K)-(N)are also designated R_(i), i=1, 2, . . . ,K, . . . , N and that links630(1)-(K) and 630(K)-(N) are also designated L_(j), j=1, 2, . . . ,K, .. . , N. This is done to allow the formulation of the equations used todescribe various embodiments of methods according to the presentinvention.

The first step towards identifying the subset of pairs of routers forwhich measurements should be made involves discovering the identity ofbackbone routers and their interconnection to enterprise routers. Indoing so, a method according to one embodiment of the present invention,such as the one given below, also partitions the collection ofenterprise routers, R_(i), i=1, 2, . . . , N, into one or moreenterprise “clusters,” C_(k), k=1, 2, . . . , m, such that eachenterprise router, R_(i), i=1, 2, . . . , N, belongs to one and only onecluster. By definition, each cluster, C_(k), k=1, 2, . . . , m, containsn_(k) number of enterprise routers, re-labeled as R_((k, j)), j=1, 2, .. . , n_(k), such that each router R_((k, j)) in the cluster, C_(k), isconnected to the backbone router, ρ_(k), using a link L_((k, j)), j=1,2, . . . , n_(k), k=1, 2, . . . , m. Such clusters may be identifiedusing a process such as that described below.

FIG. 7 is a flow diagram illustrating the operations performed in theprocess of forming “clusters” (aggregations of network elements), asdescribed above. It is noted that each cluster is identified by thedomain-name (DN) of the corresponding backbone router, forsimplification, although this need not be the case. Initialization ofthe set that will contain the collection of clusters is performed (step700). For each router (step 705), then, a different router is selected(step 710) and the sequence of network addresses along a path to theselected router is determined (step 715). Once the sequence of addresseshas been determined, a reverse domain-name look-up is performed on aname server for the first backbone router, as identified by the networkaddress of the router/port (step 720). If the domain-name of the firstbackbone router is among the domain-names corresponding to thecollection of clusters already identified, then the current router issimply added to the given cluster (step 730). If the domain-name of thefirst backbone router is not in the set of clusters (step 725), thedomain name of the new backbone router so discovered is added to the setof backbone routers (step 735). Additionally, a new cluster is createdwithin the set of clusters (step 740) and the current enterprise routeradded to the new cluster (step 745).

Once all routers have been examined thusly (for each domain-name in theset of clusters or domain-names (step 750)), several actions are thentaken. These actions include the storage of a number representing thenumber of elements in the given cluster (as denoted by the givendomain-name) (step 755); the re-labeling of the backbone router in thepresent cluster (step 760); and the re-labeling of the enterpriserouters in the present cluster (step 765). Once all of the domain-namesin the set of domain-names have been processed, the number of clustersis determined, and then stored (step 770).

Given a collection of enterprise routers, ={R_(i), i=1, 2, . . . , N},such as that presented as network 600 in FIG. 6, the grouping of routers(e.g., routers 620(1)-(K) and 620(K)-(N)) can be described in a moreformal manner. The process referred to herein as form_clusters is onesuch description.

BEGIN “form_clusters” /* Σ is a collection of clusters. */ /* Eachcluster is ID'd by the domain-name of corresponding backbone routers. */Σ = { }; For i = 1, 2, . . . , N do { Select a router, R_(j), i ≠ j; /*R_(j) may be selected arbitrarily. */ /* Determine the sequence of IPaddresses (or domain names) */ /* of routers/ports along the path torouter R_(j). */ <s₁, s₂, . . . , s_(k)> = traceroute(R_(i), R_(j)); /*μ = DN of first backbone router, ID'd by IP addr or DN of router/port,s₁ */ μ = reverse_NS_lookup(s₁); if μ ∈ Σ { then /* C_(μ) is uniquelyID'd by domain-name μ of backbone router */ add_to_cluster(R_(i),C_(μ)); else { add_to_SP_routers(μ, Σ); /* New backbone router found */C_(μ)= { }; add_to_cluster(R_(i), C_(μ)); } } k = 0 For all μ ∈ Σ do { k=k +1; n_(k) = | C_(μ) |; /* | C | denotes the number of elements in setC */ Re-label the backbone router μ as ρ_(k); Re-label routers incluster C_(μ) as R_((k,i)), i = 1, 2, . . . , n_(k); } m = k; /* m isthe number of clusters \ */ END “form_clusters”

As a result of forming clusters (and re-labeling of enterprise routers),a acceptably complete description of the interconnection betweenenterprise routers and SP-provided backbone routers is generated. Theresulting topology, corresponding to the enterprise network of FIG. 6,is illustrated in FIGS. 8A and 8B.

FIG. 8A is a block diagram illustrating the topology of a synthesizednetwork 800, depicting an inferred backbone topology generated by amethod-according to an embodiment of the present invention. Network 800includes a backbone 810, the topology and routing of which are unknownto the methods described herein, aside from the ability to detectingress router. In FIG. 8A, routers 620(1)-(K) and 620(K)-(N) aregrouped into clusters 820(1)-(m) and appear as routers 830(1,1)-(m,nm).Routers 830(1,1)-(m,nm) are coupled to backbone 810 by links840(1,1)-(m,nm), respectively. In fact, as depicted in FIG. 8A, links840(1,1)-(m,nm) couple routers 830(1,1)-(m,nm) to a number of backbonerouters 850(1)-(m). Backbone routers provide points of ingress or egressto backbone 810 for the enterprise network connected thereto. As noted,the only information that can be gleaned from backbone 810 is theexistence of backbone routers 850(1)-(m)—further information as to thetopology and routing of backbone 810 remains unknown to the methodsdescribed herein. It will be noted that routers 830(1,1)-(m,nm) are alsore-designated R_((i,j)) (i=1, 2, . . . ,k, . . . , m; j=1, 2, . . .,ni), and that links 840(1,1)-(m,nm) are also designated L_((i,j)) (i=1,2, . . . ,k, . . . , m; j=1, 2, . . . ,ni). Additionally, it will benoted that clusters 820(1)-(m) are also designated C_(i), i=1, 2, . . .,k, . . . , m, and that backbone routers 850(1)-(m) are also designatedρ_(i), i=1, 2, . . . ,k, . . . , m. This is done to allow thedescription of the parameters in terms of the equations and methodsdescribed herein.

Knowledge of topology and routing in the enterprise network may still beincomplete, however, because details concerning topology and routingwithin the SP backbone network are unavailable. This might be due, forexample, to the fact that such a backbone network may not forward probepackets (e.g., traceroute packets) beyond the ingress router. As aconsequence, discovery of the routes used within the backbone networkmay not be feasible. The backbone network may therefore be modeled as afully-connected backbone network that includes backbone routers andbackbone links.

FIG. 8B is a block diagram illustrating network 800, with routers830(1,1)-(m,nm) coupled to backbone 810 by links 840(1,1)-(m,nm),respectively, as in FIG. 8A. As before, routers 830(1,1)-(m,nm) arecoupled to backbone routers 850(1)-(m), which provide points of ingressor egress to backbone 810 for the enterprise network connected thereto.As noted, the only definitive information that can be gleaned frombackbone 810 is the existence of backbone routers 850(1)-(m)—furtherinformation as to the topology and routing of backbone 810 remainsunknown to the methods described herein. However, certain assumptionscan be made that allow the topology of backbone 810 and its routers tobe modeled. This allows delays due to actual backbone links to beidentified, or at least portions of delays to be attributed to eitherbackbone links or enterprise links.

Backbone 810 is depicted as including a number of backbone routers(i.e., backbone routers 850(1)-(m), as before). However, in FIG. 8B,each of the backbone routers is connected to each of the other backbonerouters by a backbone link (backbone links 860(1,2)-(m−1,m)). As noted,backbone links 860(1,2)-(m−1,m) are virtual, but are useful inallocating delay throughout network 800. Thus, backbone 810 isrepresented as having a fully-connected topology generated by a methodaccording to an embodiment of the present invention. As before, backbonerouters 850(1)-(m) are also designated ρ_(i), i=1, 2, . . . ,k, . . . ,m. Moreover, backbone links 860(1,2)-(m−1,m) are also designated λ(i,j)(i=1, 2, . . . ,k, . . . , (m−1); j=(i+1), . . . , m), i≠j. This is doneto allow the description of the parameters in terms of the equations andmethods described herein. The backbone network may therefore be modeledas a fully-connected network that includes routers, ρ_(k), k=1, 2, . . ., m, and links, λ_((k1,k2)), k1, k2=1, 2, . . . , m, k1≠k2.

Definitions

Several of the variables and notations discussed previously are nowdefined in light of the alternative embodiments now described thatestimate the round-trip delay and jitter for a specified set of pairs ofrouters in the case where the network includes a service-providernetwork of unknown topology and routing.

First, the user-specified requirements set, Φ, is defined as acollection of P pairs of routers for which delay and delay-jittermetrics are needed, as before. Also as before, the subset is normally asmall fraction of all possible pairs of routers. The user-specifiedrequirements set can be defined as:Φ={(π_(κ), κ=1, 2, . . . , P}  (67)whereπ_(κ)=(R _((k1,i)) , R _((k2,j)))  (68)Because round-trip delay or jitter is monitored, it is immaterialwhether the pair of routers is specified as π_(κ)=(R_((k1,i)),R_((k2,j))) or as π_(κ)=(R_((k2,j)), R_((k1,i))). Obviously, if only onedirection were measured and asymmetries existed, this would not be thecase.

Second, the measurements set, Ω, is a collection of Q number of pairs ofrouters for which round-trip delay and delay-jitter are actuallymeasured using an ANMT, for example. A fundamental property of themeasurements set is that, based on measurements so obtained, one canaccurately estimate delay/jitter metrics for all pairs of routersspecified in the requirements set, Φ. In particular, the measurementsset can be defined asΩ={π′_(κ), κ=1, 2, . . . , Q}  (69)whereπ′_(κ)=(R _((k1,i)) , R _((k2,j)))  (70)As will be apparent to one of skill in the art, Ω will always be asubset of Φ.

It is assumed that, for a given pair of enterprise routers, R_((k1,i))and R_((k2,j)), the only available route between them is through thebackbone routers, ρ_(k1) and π_(k2), that physically connect R_((k1,i))and R_((k2,j)) to the SP backbone, respectively. As a result, the routebetween them is given byLink_path(R _((k1,i)) , R _((k2,j)))=<L _((k1,i)), λ_((k1,k2)) , L_((k2,j))>  (71)Three types of delay components that contribute to the round-trip delaybetween a pair of enterprise routers may be defined:

-   -   1. d_((k,i)), k=1, 2, . . . , m, i=1, 2, . . . , n_(k), the        round-trip delay in transferring an IP packet between the pair        of routers, R_((k,i)) and ρ_(k), over the link, L_((k,i));    -   2. s_((k,i)), k=1, 2, . . . , m, i=1, 2, . . . , n_(k), the time        it takes to generate, receive and process a probe packet within        transport and application layers in an enterprise router,        R_((k,i)); and    -   3. c_((k1, k2)), k1, k2=1, 2, . . . , m, k1≠k2, the round-trip        delay in transferring an IP packet between the pair of backbone        routers, ρ_(k1) and ρ_(k2).        This definition builds on that previously given with regard to        protocol transfer delay and processing delay.

The ordered collection of delay components is re-written in the form ofa vector of size 2N+m*(m−1)/2: $\quad\begin{matrix}\begin{matrix}{{x^{T} = ⁠\begin{matrix}\left\lbrack d_{({1,1})} \right. & d_{({1,2})} & \ldots & d_{({1,{n1}})}\end{matrix}}\quad} \\{\begin{matrix}d_{({2,1})} & d_{({2,2})} & \ldots & d_{({2,{n2}})}\end{matrix}} \\{\begin{matrix}\ldots & \quad & \quad & \quad\end{matrix}} \\{\begin{matrix}d_{({m,1})} & d_{({m,2})} & \ldots & d_{({m,{nm}})}\end{matrix}} \\{\begin{matrix}c_{({1,2})} & c_{({1,3})} & \ldots & c_{({1,{rn}})}\end{matrix}} \\{\begin{matrix}c_{({2,3})} & c_{({2,4})} & \ldots & c_{({2,{rn}})}\end{matrix}} \\{\begin{matrix}\ldots & \quad & \quad & \quad\end{matrix}} \\{\begin{matrix}c_{({{m - 1},m})} & \quad & \quad & \quad\end{matrix}} \\{\begin{matrix}s_{({1,1})} & s_{({1,2})} & \ldots & s_{({1,{n1}})}\end{matrix}} \\{\begin{matrix}s_{({2,1})} & s_{({2,2})} & \ldots & s_{({2,{n2}})}\end{matrix}} \\{\begin{matrix}\ldots & \quad & \quad & \quad\end{matrix}} \\\left. \begin{matrix}s_{({m,1})} & s_{({m,2})} & \ldots & s_{({m,{nm}})}\end{matrix} \right\rbrack\end{matrix} & (72)\end{matrix}$

In view of the definition of delay components set out above, theround-trip delay between a given pair of enterprise routers, R_((k1,i))and R_((k2,j)), as seen by network monitoring applications, is given by:

-   -   (a) in the case where k1≠k2, and assuming that        link_path(R_((k1,i)), R_((k2,j)))=<L(_(k1,i)), λ_((k1,k2)),        L_((k2,j)), $\begin{matrix}        {z_{\kappa} = {{Delay}\left( {R_{({{k1},i})},R_{({{k2},j})}} \right)}} & {\quad\left( {73a} \right)} \\        {= {s_{({{k1},i})} + d_{({{k1},i})} + c_{({{k1},{k2}})} + d_{({{k2},j})} + s_{({{k2},j})}}} & {\quad\left( {73b} \right)} \\        {= {F_{\kappa}x}} & {\quad\left( {73c} \right)}        \end{matrix}$    -    where        F _(κ)=[0 . . . 0 1 0 . . . 0 1 0 . . . 0 1 0 . . . 0 1 0 . . .        0 1 0 . . . 0]  (73d)    -    is a row vector of size 2N+m*(m−1)/2, and the 1's in F_(κ)        appear in columns corresponding to column variables,        c_((k1,k2)), d_((k1,i)), s_((k1,i)), d_((k2,j)), and s_((k2,j));    -   (b) in the case where k1=k2, and assuming that path(R_((k1,i)),        R_((k2,j)))=<L_((k1,i)), L_((k1,j))>, $\begin{matrix}        {z_{\kappa} = {{Delay}\left( {R_{({{k1},i})},R_{({{k1},j})}} \right)}} & {\quad\left( {74a} \right)} \\        {= {s_{({{k1},i})} + d_{({{k1},i})} + d_{({{k1},j})} + s_{({{k1},j})}}} & {\quad\left( {74b} \right)} \\        {= {F_{\kappa}x}} & {\quad\left( {74c} \right)}        \end{matrix}$    -    where        F _(κ)=[0 . . . 0 1 0 . . . 0 1 0 . . . 0 1 0 . . . 0 1 0 . .        0]  (74d)    -    and the 1's in F_(κ) appear in columns corresponding to column        variables, d_((k1,i)), s_((k1,j)), d_((k1,j)), and s_((k1,j)).

Because delay, z_(κ)=Delay(R_((k1,i)), R_((k2,j))), varies, one maydefine delay-jitter between a pair of routers, R_((k1,i)) and R_((k2,j))as follows:Delay-Jitter(R _((k1,i)) , R _((k2,j)))=√E{(z _(κ) −E{z _(κ)})²}  (75)where z_(κ)=Delay(R_((k1,i)), R_((k2,j))) and E{.} is the expectationoperation. The round-trip delay between the specified pairs of routersin Φ={π_(κ)=(R_((k1,i)), R_((k2,j))), κ=1, 2, . . . , P}, may bere-written as a vector equation:z=Fx  (76)where $\begin{matrix}{\begin{matrix}{z = \begin{bmatrix}z_{1} & \quad\end{bmatrix}} \\{\begin{bmatrix}z_{2} & \quad\end{bmatrix}} \\{\begin{bmatrix}\ldots & \quad\end{bmatrix}} \\{\begin{bmatrix}z_{p} & \quad\end{bmatrix}}\end{matrix}{and}} & (77) \\\begin{matrix}{F = \begin{bmatrix}F_{1} & \quad\end{bmatrix}} \\{\begin{bmatrix}F_{2} & \quad\end{bmatrix}} \\{\begin{bmatrix}\ldots & \quad\end{bmatrix}} \\{\begin{bmatrix}F_{p} & \quad\end{bmatrix}}\end{matrix} & (78)\end{matrix}$

The P×2N+m*(m−1)/2 matrix, F, helps determine the subset of pairs ofrouters between which delay measurements are necessary. If F₁, F₂, . . ., F_(Q) is indeed a maximal set of linearly independent rows of F, thenrow vectors, F_(Q+1), F_(Q+2), . . . , F_(P) can be expressed as alinear combination of F₁, F₂, . . . , F_(Q) as follows:F _(κ)=Σ_(i−1, . . . ,Q)(α_(κ,i) F _(i)), κ=Q+1, Q+2, . . . , P  (79)

The constants α_(κ,i) may be re-organized in the form of a row vector(of size Q) as follows:α_(κ)=[α_(κ,1), α_(κ,2), . . . , α_(κ,Q) ], κ=Q+1, Q+2, . . . , P  (80)while the P×Q matrix, A, is defined as: $\quad\begin{matrix}\begin{matrix}{A = \begin{bmatrix}I & \quad\end{bmatrix}} \\{\begin{bmatrix}\alpha_{Q + 1} & \quad\end{bmatrix}} \\{\begin{bmatrix}\alpha_{Q + 2} & \quad\end{bmatrix}} \\{\begin{bmatrix}\ldots & \quad\end{bmatrix}} \\{\begin{bmatrix}\alpha_{p} & \quad\end{bmatrix}}\end{matrix} & (81)\end{matrix}$where the matrix, I, is a Q×Q identity matrix.A Measurement and Estimation Scheme for a Specified Requirements Set

A procedure according to one embodiment of the present invention is nowdescribed that computes round-trip delay and jitter for every pair ofrouters, π_(κ)=(R_((k1,i)), R_((k2,j))) contained in a specifiedrequirements set, Φ, using measurements between pairs of routers in themeasurements set, Ω, and identified using a method such as thatdescribed below. The scheme is particularly useful when the number ofpairs of routers in the requirements set is not large.

FIG. 9 is a flow diagram illustrating a process that computes round-tripdelay and jitter for every pair of routers contained in a specifiedrequirements set using measurements between router pairs in themeasurements set. The computation of round-trip delay and jitter (alsoreferred to herein as delay-jitter) begins with the specification of therequirements set (step 900). Next, the delay-components vector (x) isconstructed per Equation (72) and associated equations (step 905). Foreach router pair in the specified requirement set (step 910), thefollowing actions are performed. First, the given router pair isidentified as being an element of the collection of router pairs (step912). Next, path information is obtained for the given pair of routers(step 914). Finally, the row vector (F_(k)) is then determined (step916).

Once these steps have been carried out for each router pair in thespecified collection of router pairs, the F matrix is constructed inaccordance with Equation (78) and associated equations (step 920). Oncethe F matrix has been constructed, a maximal set of independent rows ofthe F matrix are identified (step 925). Once identified, the number ofindependent rows in the F matrix is stored (Step 930). The rows of the Fmatrix (as well as the router pairs in Φ) are reordered such that thefirst Q rows of the F matrix are independent (step 935). For eachdependent row of the F matrix (step 940), the constants used to expressthe dependent row in terms of the independent rows are determined (step942), and a corresponding row vector is constructed for these constants(step 944). Then the first Q pairs of routers (representing the first Qindependent rows of the F matrix) are copied in order to form the subsetΩ, representing the measurements set (step 945). As has been noted, thequantity Q represents the number of rows in the maximal set ofindependent rows of the F matrix.

At this point, the values for the start time and measurement intervalare set (step 950). Next, while the measurement process remainsunfinished (step 955), the following actions are performed. For eachindependent row of F (step 958), the average round-trip delay and jitterbetween pairs of routers in Ω are measured during the time interval(step 960). Next, for each router pair in Φ (step 965), the round-tripdelay vector is estimated (step 970). In a similar fashion, for eachrouter pair in Φ (step 975), the round-trip delay-jitter is computed(step 980). This completes the exemplary process of computing round-tripdelay and jitter for the routers contained in the specified requirementset.

In using the above method, the measurement error is assumed to benegligible. Moreover, delay estimates are optimal provided no otherdelay measurements or a priori estimates are available. The process ofcomputing round-trip delay and jitter for every pair of routerscontained in a specified requirements set using measurements betweenrouter pairs in the measurements set is now outlined using a formalrepresentation.

BEGIN /* Specify collection of router pairs in requirement set. */ /*Such a pair of router should not be, for example, (R_((k1,i)), ρ_(k2))*/ Φ = {π_(κ) = (R_(k1,i)), R_((k2,j))), κ = 1, 2, . . . , P}; Constructdelay-components vector, x; /* see Eqn (72) and associated equations */For each κ = 1, 2, . . . , P, do { Identify the pair of routers π_(κ) =(R_((k1,i)), R_((k2,j))) ∈ Φ; Obtain link_path(R_((k1,i)), R(_((k2,j)))= <L_((k1,1)), λ_((k1,k2)), L_((k2,j))>; Determine the row vector,F_(k), such that z_(κ) = Delay(R_((k1,i)), R_((k2,j))) = F_(κ) x };Construct P × 2N+m*(m−1)/2 matrix, F; /* see Eqn (78) and associatedequations */ Identify a maximal set of independent rows of F; Q = thenumber of maximal set of independent rows of F; Reorder rows of F androuter pairs in Φ such that the first Q rows, F₁, F₂, . . . , F_(Q), areindependent; For each κ = Q+1, Q+2, . . . , P, do{ Determine α_(κ,1),α_(κ,2), . . . , α_(κ,Q)such that F_(κ) = Σ_(i=1,...,Q)(α_(κ,i) F_(i));Construct the row vector α_(κ) = [α_(κ,1), α_(κ,2), . . . , α_(κ,Q)] };Copy the first Q pairs of routers from Φ to form the subset Ω; Identifythe start time t₀, and the measurement interval, dt; While (!finished) {For each κ = 1, 2, . . . , Q, do { During the time interval (t₀, t₀ +dt)use a network monitoring tool to measure average round-trip delay,y_(κ), and delay-jitter, γ_(κ), between the pair of routers, π_(κ) =(R_((k1,i)), R_((k2,j))) For each pair of routers in Φ, estimateround-trip delay vector, Δ_(κ) For each κ = 1, 2, . . . , Q, do { Δ_(κ)= y_(κ) }; For each κ = Q+1, Q+2, . . . , P, do { Δ_(κ) =Σ_(i=1,...,Q)(α_(κ,i) y_(i)) }; For each pair of routers in Φ, computeround-trip delay-jitter, σ_(κ), For each κ = 1, 2, . . . , Q, do { σ_(κ)= γ_(κ) }; For each κ = Q+1, Q+2, . . . , P, do { σ_(κ) = √Σ_(i=1,...,Q)(α_(κ,i) ² γ_(i) ²) }; } END

The above method may be improved by recognizing that the variables,s_((k1,i)) and d_((k1,i)), always appear together ass_((k1,i))+d_((k1,i)). In that case, the variable x may be redefined asfollows (see equation also (72) and associated equations):$\quad\begin{matrix}\begin{matrix}{{x^{T} = ⁠\begin{matrix}\left\lbrack {d_{({1,1})} + s_{({1,1})}} \right. & {d_{({1,2})} + s_{({1,2})}} & \ldots & {d_{({1,{n1}})} + s_{({1,{n1}})}}\end{matrix}}\quad} \\{\begin{matrix}{d_{({2,1})} + s_{({2,1})}} & {d_{({2,2})} + s_{({2,2})}} & \ldots & {d_{({2,{n2}})} + s_{({2,{n2}})}}\end{matrix}} \\{\begin{matrix}\ldots & \quad & \quad & \quad\end{matrix}} \\{\begin{matrix}{d_{({m,1})} + s_{({m,1})}} & {d_{({m,2})} + s_{({m,2})}} & \ldots & {d_{({m,{nm}})} + s_{({m,{mn}})}}\end{matrix}} \\{\begin{matrix}c_{({1,2})} & c_{({1,3})} & \ldots & c_{({1,m})}\end{matrix}} \\{\begin{matrix}c_{({2,3})} & c_{({2,4})} & \ldots & c_{({2,m})}\end{matrix}} \\{\begin{matrix}\ldots & \quad & \quad & \quad\end{matrix}} \\\left. c_{({{m - 1},m})} \right\rbrack\end{matrix} & (82)\end{matrix}$As a result, one may redefine D_((i,j))=d_((i,j))+s_((i,j)) for all iand j, and represent delay, Delay(R_((k1,I)), R_((k2,j))), in terms ofvector, x, as follows:

-   -   (a) in the case where k1≠k2, $\begin{matrix}        {z_{\kappa} = {{Delay}\left( {R_{({{k1},i})},R_{({{k2},j})}} \right)}} & {\quad\left( {83a} \right)} \\        {= {D_{({{k1},i})} + c_{({{k1},{k2}})} + D_{({{k2},j})}}} & {\quad\left( {83b} \right)} \\        {= {F_{\kappa}x}} & {\quad\left( {83c} \right)}        \end{matrix}$    -    where        F _(κ)=[0 . . . 0 1 0 . . . 0 1 0 . . . 0 1 0 . . . 0]  (83d)    -    is a row vector of size N+m*(m−1)/2, and the 1's in F_(κ)        appear in columns corresponding to column variables, D_((k1,i)),        D_((k2,j)) and c_((k1, k2)),    -   (b) in the case where k1≠k2, $\begin{matrix}        {z_{\kappa} = {{Delay}\left( {R_{({{k1},i})},R_{({{k1},j})}} \right)}} & {\quad\left( {84a} \right)} \\        {= {D_{({{k1},i})} + D_{({{k1},j})}}} & {\quad\left( {84b} \right)} \\        {= {F_{\kappa}x}} & {\quad\left( {84c} \right)}        \end{matrix}$    -    where        F _(κ)=[0 . . . 0 1 0 . . . 0 1 0 . . . 0]  (84d)    -    is a row vector of size N+m*(m−1)/2, and the ones in F_(κ)        appear in columns corresponding to variables D_((k1,i)) and        D_(k1,j)).        An Exemplary Network

FIG. 10A is a block diagram illustrating an exemplary network 1000,where N=7. Network 1000 includes a backbone 1010 and routers1020(1)-(7). Routers 1020(1)-(7) are coupled to backbone 1010 by links1030(1)-(7), respectively. It will be noted that routers 1020(1)-(7) arealso designated R_(i), i=1, 2, . . . ,7, and that links 1030(1)-(7) arealso designated L_(j), j=1, 2, . . . ,7, in order to allow network 1000to be discussed in terms of the parameters discussed herein. The onlyinformation that can be gleaned from backbone 1010 is the existence ofbackbone routers that allow ingress of network traffic into backbone1010—further information as to the topology and routing of backbone 1010remains unknown. It is desirable to determine, or at least estimate, thedelay experienced by network traffic flowing through the enterprisenetwork, including that transiting backbone 1010. This information maybe derived using the techniques just described, and results in thescenario depicted in FIG. 10B.

FIG. 10B illustrates exemplary network 1000 after being apportioned intoclusters using a method according to one embodiment of the presentinvention. As before, network 1000 includes routers 1020(1)-(7) coupledto backbone 1010 by links 1030(1)-(7), respectively. It is understoodthat backbone routers provide points of ingress or egress to backbone1010 for the enterprise network connected thereto. As noted previously,the existence of backbone routers 1050(1)-(4) can be deduced from theinteraction of backbone routers 1050(1)-(4) with routers 1020(1)-(7),but further information as to the topology and routing of backbone 1010remains unknown, as noted.

In order to allow the determination of delay and jitter throughoutnetwork 1000, embodiments according to the present invention apportionthe enterprise network into clusters using a method of cluster formationsuch as that described previously. Using such a method, the enterprisenetwork can be modeled as having m=4 clusters. These are depicted inFIG. 10B as clusters 1040(1)-(4), and are also designated as clustersC₁, C₂, C₃, C₄, in order to facilitate the discussions herein. Backbone1010 is depicted as including a number of backbone routers (i.e.,backbone routers 1050(1)-(4)), with the routers in each of clusters1040(1)-(4) coupled to one of the a corresponding backbone routers1050(1)-(4). Moreover, each of backbone routers 1050(1)-(4) is connectedto each of the other ones of backbone routers 1050(1)-(4) by a givenbackbone link (backbone links 1060(1,2)-(3,4)). As noted, backbone links1060(1,2)-(3,4) are virtual, but are useful in allocating delay andjitter throughout network 1000. Thus, backbone 1010 is represented ashaving a fully-connected topology generated by a method according to anembodiment of the present invention. As before, backbone routers1050(1)-(7) are also designated ρ_(i), i=1, 2, . . . , 4. Moreover,backbone links 1060(1,2)-(3,4) are also designated λ(i,j) (i=1, . . . ,3; j=i+1, . . . , 4). This is done to allow the description of theparameters in terms of the equations and methods described herein. Thebackbone network may therefore be modeled as a fully-connected networkthat includes routers, ρ_(k), k=1, 2, . . . , 4, and links, λ_((k1,k2)),k1, k2=1, 2, . . . , 4, k1≠k2. It will also be noted that R_(i), i=1, 2,. . . , 7, have been renumbered to reflect their designations in thefollowing calculations and appear as R_((1,1)), R_((1,2)), R_((1,3)),R_((2,1)), R(_(3,1)), R_((4,1)) and R_((4,2)), respectively.

As can be seen, there is a unique route between any given pair ofenterprise routers. In particular, the route between R_((k1,i)) andR_((k2,j)) is given by:Link_path(R _((k1,i)) , R _((k2,j)))=<L _((k1,i)), λ_((k1, k2)) , L_((k2,j))>  (85)The execution of the method to estimate round-trip delay and jitter forthe specified requirements set, Φ, is now illustrated. Let$\quad\begin{matrix}\begin{matrix}{\Phi = \left\{ {\left( {R_{({2,1})},R_{({1,1})}} \right),\left( {R_{({2,1})},R_{({1,2})}} \right),\left( {R_{({2,1})},R_{({1,3})}} \right),} \right.} \\{\left( {R_{({3,1})},R_{({1,1})}} \right),\left( {R_{({3,1})},R_{({1,2})}} \right),\left( {R_{({3,1})},R_{({1,3})}} \right),} \\{\left( {R_{({4,1})},R_{({1,1})}} \right),\left( {R_{({4,1})},R_{({1,2})}} \right),\left( {R_{({4,1})},R_{({1,3})}} \right),} \\\left. {\left( {R_{({4,2})},R_{({1,1})}} \right),\left( {R_{({4,2})},R_{({1,2})}} \right),\left( {R_{({4,2})},R_{({1,3})}} \right)} \right\}\end{matrix} & (86)\end{matrix}$The delay-component vector, $\quad\begin{matrix}\begin{matrix}{x^{T} = \begin{bmatrix}D_{({1,1})} & \quad\end{bmatrix}} \\{\begin{bmatrix}D_{({1,2})} & \quad\end{bmatrix}} \\{\begin{bmatrix}D_{({1,3})} & \quad\end{bmatrix}} \\{\begin{bmatrix}D_{({2,1})} & \quad\end{bmatrix}} \\{\begin{bmatrix}D_{({3,1})} & \quad\end{bmatrix}} \\{\begin{bmatrix}D_{({4,1})} & \quad\end{bmatrix}} \\{\begin{bmatrix}D_{({4,2})} & \quad\end{bmatrix}} \\{\begin{bmatrix}c_{({1,2})} & \quad\end{bmatrix}} \\{\begin{bmatrix}c_{({1,3})} & \quad\end{bmatrix}} \\{\begin{bmatrix}c_{({1,4})} & \quad\end{bmatrix}} \\{\begin{bmatrix}c_{({2,3})} & \quad\end{bmatrix}} \\{\begin{bmatrix}c_{({2,4})} & \quad\end{bmatrix}} \\{\begin{bmatrix}c_{({3,4})} & \quad\end{bmatrix}}\end{matrix} & (87)\end{matrix}$Because z₁=Delay(R_((2,1)), R_((1,1)))=D_((1, 1))+c_((1, 2))+D_((2, 1)),F ₁=[1 0 0 1 0 0 0 1 0 0 0 0 0]  (88)Similarly, one may obtain F_(i), i=2, 3, . . . , 12. The resulting 12×13matrix F is given by:

D_((1,1)) D_((1,2)) D_((1,3)) D_((2,1)) D_((3,1)) D_((4,1)) D_((4,2))c_((1,2)) c_((1,3)) c_((1,4)) c_((2,3)) c_((2,4)) c_((3,4)) Z₁ 1 1 1 Z₂1 1 1 Z₃ 1 1 1 Z₄ 1 1 1 Z₅ 1 1 1 Z₆ 1 1 1 Z₇ 1 1 1 Z₈ 1 1 1 Z₉ 1 1 1 Z₁₀1 1 1 Z₁₁ 1 1 1 Z₁₂ 1 1 1It can be verified that F₁, F₂, F₃, F₄, F₇, F₁₀ are independent rowvectors. As a result, Q=6. The row vectors, F_(k) are re-ordered, asthose corresponding to z₁, z₂, z₃, z₄, z₇, z₁₀. The re-ordered versionof F is:

D_((1,1)) D_((1,2)) D_((1,3)) D_((2,1)) D_((3,1)) D_((4,1)) D_((4,2))c_((1,2)) c_((1,3)) c_((1,4)) c_((2,3)) c_((2,4)) c_((3,4)) Z₁ 1 1 1 Z₂1 1 1 Z₃ 1 1 1 Z₄ 1 1 1 Z₇ 1 1 1 Z₁₀ 1 1 1 Z₅ 1 1 1 Z₆ 1 1 1 Z₈ 1 1 1 Z₉1 1 1 Z₁₁ 1 1 1 Z₁₂ 1 1 1Further, it can be shown thatF ₅ =F ₂ +F ₄ −F ₁  (89a)F ₆ =F ₃ +F ₄ −F ₁  (89b)F ₈ =F ₂ +F ₇ −F ₁  (89c)F ₉ =F ₃ +F ₇ −F ₁  (89d)F ₁₁ =F ₂ +F ₁₀ −F ₁  (89e)F ₁₂ =F ₃ +F ₁₀ −F ₁  (89f)Consequently,α₅=[−1 1 0 1 0 0]  (90a)α₆=[−1 0 1 1 0 0]  (90b)α₈=[−1 1 0 0 1 0]  (90c)α₉=[−1 0 1 0 1 0]  (90d)α₁₁=[−1 1 0 0 0 1]  (90e)α₁₂=[−1 0 1 0 0 1]  (90f)As a result $\quad\begin{matrix}\begin{matrix}{\Omega = \left\{ {\left( {R_{({2,1})},R_{({1,1})}} \right),\left( {R_{({2,1})},R_{({1,2})}} \right),} \right.} \\{\left( {R_{({2,1})},R_{({1,3})}} \right),\left( {R_{({3,1})},R_{({1,1})}} \right),} \\\left. {\left( {R_{({4,1})},R_{({1,1})}} \right),\left( {R_{({4,2})},R_{({1,1})}} \right),} \right\}\end{matrix} & (91)\end{matrix}$Initiallyt ₀=0 (seconds)  (92)anddt=60 (seconds)  (93)which can then be used to define an interval (t₀, t₀+dt). During theinterval (t₀, t₀+dt), a network monitoring tool is used to measureround-trip delay and jitter between pairs of routers identified by Ω, toyield delay measurements, y₁, y₂, y₃, y₄, y₇, y₁₀, and delay-jittermeasurements, γ₁, γ₂, γ₃, γ₄, γ₇, γ₁₀. An estimate of delay betweenpairs of routers in Φ is given byΔ₁ =y ₁  (94a)Δ₂ =y ₂  (94b)Δ₃ =y ₃  (94c)Δ₄ =y ₄  (94d)Δ₇ =y ₇  (94e)Δ₁₀ =y ₁₀  (94f)andΔ₅ =y ₂ +y ₄ −y ₁  (95a)Δ₆ =y ₃ +y ₄ −y ₁  (95b)Δ₈ =y ₂ +y ₇ −y ₁  (95c)Δ₉ =y ₃ +y ₇ −y ₁  (95d)Δ₁₁ =y ₂ +y ₁₀ −y ₁  (95e)Δ₁₂ =y ₃ +y ₁₀ −y ₁  (95f)An estimate of delay-jitter between pairs of routers in Φ is given byσ₁=γ₁  (96a)σ₂=γ₂  (96b)σ₃=γ₃  (96c)σ₄=γ₄  (96d)σ₇=γ₇  (96e)σ₁₀=γ₁₀  (96f)andσ₅=√(γ₂ ²+γ₄ ²+γ₁ ²)  (97a)σ₆=√(γ₃ ²+γ₄ ²+γ₁ ²)  (97b)σ₈=√(γ₂ ²+γ₇ ²+γ₁ ²)  (97c)σ₉=√(γ₃ ²+γ₇ ²+γ₁ ²)  (97d)σ₁₁=√(γ₂ ²+γ₁₀ ²+γ₁ ²)  (97e)σ₁₂=√(γ₃ ²+γ₁₀ ²+γ₁ ²)  (97f)Once complete, the steps of delay measurement and estimation can berepeated by incrementing t₀ (e.g., t₀=t₀+dt).IV. Estimating Delay and Jitter in an Enterprise Network Coupled to aBackbone Having an Unknown Topology and Estimating Delay Componentswithin the Backbone

A measurement scheme according to one embodiment of the presentinvention is now described that allows one to not only estimate thedelay between a given pair of enterprise routers but, more importantly,to apportion the same to various links along the route, including theSP-provided backbone. This allows the determination of whethercongestion (if any) exists within the backbone or over the local (orremote) links. It is assumed that a model of the enterprise network isavailable. That is, an enterprise network exists and consists ofenterprise routers that are interconnected through a backbone networkprovided by a service provider (as in FIG. 8A). It is further assumedthat a model of the backbone network in terms of backbone routers andvirtual links (as in FIG. 8B) has been obtained using methods previouslydescribed. As such, such a network is modeled as a collection of mclusters of enterprise routers, with a backbone router corresponding toeach.

FIG. 11 is a flow diagram illustrating the operations performed in theprocess of apportioning delay components. The apportioning of delaycomponents begins with the specification of a router pair between whichdelay and jitter are to be monitored (step 1100). Once the routers ofthe router pair are specified, the clusters corresponding to theserouters are then identified (step 1105) and the size of those clustersis determined (step 1110). The size of the clusters are designated as n1and n2, corresponding to the cluster designated as the first cluster andthe cluster designated as the second cluster, respectively (step 1115).The designation of a cluster as the first or second cluster isarbitrary, and is driven by the need to identify the clusters with oneof the following possible scenarios. If the size of both clusters isgreater that two (i.e., each cluster contains at least two enterpriserouters) (step 1120), the process designated as Sub_2_2 is employed(step 1130). If the size of the first cluster is at least 2, but thesize of the second cluster is one (step 1140), the process designated asSub_2_1 is employed (step 1150). If both clusters have a size of one(step 1160), the process designated as Sub_1_1 is employed (step 1170).It will be noted that the processes just referred to are designatedusing the minimum sizes for the first and second clusters. If none ofthe aforementioned cases hold, and error condition exists and isindicated as such (step 1180).

The overall paradigm for apportioning delay components is now outlinedusing a formal representation. The process referred to herein asApportion_delay_components is one such description.

BEGIN “Apportion_delay_components” /* Delay and jitter components aremonitored between these two enterprise routers. */ Specify a pair ofrouters, (R_((k1,i)), R_((k2,j))), k1 ≠ k2; Identify the correspondingclusters, C_(k1) and C_(k2); Identify the size, n1 and n2, of thecorresponding clusters; /* Note that n1 and n2 can be selected such thatn1 ≧ n2 without loss of generality */ /* Three cases can be identified:*/ /* The size of both the first and second clusters are greater than orequal to 2 */ If n1 ≧ 2, n2 ≧ 2 then { Sub_2_2 ( ) }; /* The size of thefirst cluster is greater than or equal to 2 */ /* The size of the secondclusters is equal to 1 */ If n1 ≧ 2, n2 = 1 then { Sub_2_1 ( ) }; /* Thesize of both the first and second clusters are equal to 1 */ If n1 = 1,n2 = 1 then { Sub_1_1 ( ) } END “Apportion_delay_components”

FIG. 12 is a flow diagram illustrating the operations performed in theprocess of apportioning delay components when the size of both a firstcluster and a second cluster are greater than or equal to two. Theprocess begins with the identification of the router pair (step 1200)and the definition of the measurements set (step 1210). Next, thedelay-components vector is constructed (step 1220). The start time andmeasurement interval are then set (steps 1230 and 1240). While themeasurement of delay and jitter has not been completed (step 1250), thefollowing steps are carried out. For each entry in the measurements set(step 1255), and while within the time interval (step 1260), the averageround-trip delay and delay-jitter between the given router pair ismeasured (step 1265). Once such measurements have been completed foreach entry in the measurements set during the time interval, an estimateof round-trip delay, in the form of a round-trip delay vector, iscomputed (step 1270). In a similar fashion, an estimate of round-tripjitter, in the form of a round-trip jitter vector, is computed (step1280). Once these computations are completed, the start time may beincremented (step 1290) and the process repeated for the desired numberof iterations.

A method for apportioning delay components when the given pair ofrouters, (R_((k1,i)), R_((k2,j))), both have sizes larger than or equalto two (i.e., n1=|C_(k1)|≧2, n2=|C_(k2)|≧2), is now outlined using aformal representation. This process is referred to herein as Sub_2_2.

BEGIN “Sub_2_2” Identify routers, R_((k1,i+1)), R_((k2,j+1)); /* If i =n1, then let i+1 = 1.    If j = n2, then let j+1 = 1. */ /* Define themeasurements set */ Ω = { π₁ = (R_((k1,i)), R_((k1,i+1))), π₂ =(R_((k1,i)), R_((k2,j))), π₃ = (R_((k2,j)), R_((k1,i+1))), π₄ =(R_((k2,j)), R_((k2,j+1))), π₅ = (R_((k1,i)), R_((k2,j+1))) }; /* Notethat D_((k1,i)) = d_((k1,i)) + s_((k1,i)) */ /* Construct thedelay-components vector */ x^(T) = [D_((k1,i)), D_((k1,i+1)),c_((k1,k2)), D_((k2,j)), D_((k2,j+1))]; t₀ = start_time; dt =measurement_interval; While (!finished) { For each κ = 1 to 5, do {While within time interval (t₀, t₀ +dt) { Use a network monitoring toolto measure average round- trip delay, y_(κ), and delay-jitter, γ_(κ),between the pair of routers, π_(κ), in set Ω. } }; Compute an estimateof link-level round-trip delay vector, x, using δ = [ 0.5 0.5 −0.5 0 0] [y₁] [ 0.5 −0.5 0.5 0 0 ] [y₂] [ −0.5 0 0.5 −0.5 0.5 ] [y₃] [ 0 0.5 00.5 −0.5 ] [y₄] [ 0 −0.5 0 0.5 0.5 ] [y₅]; Compute an estimate oflink-level round-trip jitter vector, x, using Σ = [ 0.25 0.25 0.25 0 0] [γ₁ ²] [ 0.25 0.25 0.25 0 0 ] [γ₂ ²] [ 0.25 0 0.25 0.25 0.25 ] [γ₃ ²][ 0 0.25 0 0.25 0.25 ] [γ₄ ²] [ 0 0.25 0 0.25 0.25 ] [γ₅ ²] where Σ^(T)= [σ² _((k1,i)), σ² _((k1,i+1)), σ² _((k1,k2)), σ² _((k2,j)), σ²_((k2,j+1))]; t₀ = t₀+dt } END “Sub_2_2”

FIG. 13 is a flow diagram illustrating the operations performed in theprocess of apportioning delay components when the size of a firstcluster is greater than or equal to two and the size of a second clusteris equal to one. The process begins with the identification of a givenrouter pair (step 1300) and the definition of the measurement set (step1310). Next, a delay-components vector is constructed (step 1320). Thestart time and measurement interval are then set (steps 1330 and 1340).While further measurements are desired (step 1350), the followingactions are performed. For each entry in the measurements set (step1355), and while within the time interval (step 1360), the averageround-trip delay and delay-jitter between the given router pair aremeasured (step 1365). Once each entry in the measurement set is analyzedduring the given time interval, an estimate of round-trip delay,represented by a round-trip delay vector, is computed (step 1370). In asimilar fashion, an estimate of round-trip delay-jitter is computed(step 1380). The start time is incremented (step 1390) and furtheriterations are performed, as desired.

The method for apportioning delay components, when the size of a firstcluster (n1) is greater than or equal to two and the size of a secondcluster (n2) is equal to one, is now outlined using a formalrepresentation. Thus, for the given pair of routers, (R_(k1,i)),R_(k2,j))), n1=|C_(k1)|≦2, n2=|C_(k2)|=1. This process is referred toherein as Sub_2_1.

BEGIN “Sub_2_1” Identity routers, R_((k1,j+1)) /* If i = n1, then leti+1 = 1. */ /* Define the measurements set */ Ω = {π₁ = (R_((k1,i)),R_((k1,i+1))), π₂ = (R_((k1,i)), R_((k2,j))), π₃ = (R_((k2,j)),R_((k1,i+1)))}; /* Construct the delay-components vector */ x^(T) =[D_((k1,i)), D_((k1,i+1)), c_((k1,k2))+D_((k2,j))]; t₀ = start_time; dt= measurement_interval; While (!finished) { For each κ = 1 to 3, do {While within time interval (t₀, t₀ +dt) { Use a network monitoring toolto measure average round- trip delay, y_(κ), and delay-jitter, γ_(κ),between the pair of routers, π_(κ), in set Ω. } }; Compute an estimateof link-level round-trip delay vector, x, using δ= [ 0.5 0.5 −0.5 ] [y₁][ 0.5 −0.5 0.5 ] [y₂] [ −0.5 0.5 0.5 ] [y₃]; Compute an estimate oflink-level round-trip jitter vector, x, using Σ = [ 0.25 0.25 0.25 ] [γ₁²] [ 0.25 0.25 0.25 ] [γ₂ ²] [ 0.25 0.25 0.25 ] [γ₃ ²] where Σ^(T) = [σ²_((k1,i)), σ² _((k1,i+1)), σ² _((k1,k2))+σ² _((k2,j))]; t₀ = t₀+dt } END“Sub_2_1”

FIG. 14 is a flow diagram illustrating the operations performed in theprocess of apportioning delay components when the size of both the firstcluster and the second cluster is equal to one. The process begins withthe definition of the measurement set (step 1400) and the constructionof the delay-components vector (step 1410). Next, the start time andmeasurement interval are set (steps 1420 and 1430). While moremeasurements are to be taken (step 1440), and while within the timeinterval (step 1450), the average round-trip delay and delay-jitterbetween the given router pair are measured (step 1460). Once themeasurements with in the time interval have been taken, an estimate ofthe round-trip delay, represented by the round-trip delay vector, iscomputed (step 1470). In similar fashion, an estimate of the round-tripdelay-jitter is computed, resulting in a round-trip delay-jitter vector(step 1480). Once the afar mentioned computations have been made, thestart time is incremented in order to take a crash set of measurements(step 1490).

The method for apportioning delay components, when the size of a firstcluster (n1) and the size of a second cluster (n2) are equal to one, isnow outlined using a formal representation. Thus, for the given pair ofrouters, (R_((k1,i)), R_((k2,j))), n1=|C_(k1)|=1, n2=|C_(k2)|=1. Thisprocess is referred to herein as Sub_1_1.

BEGIN “Sub_1_1” Ω = {π₁ = (R_((k1,i)), R_((k2,j)))}; Construct thedelay-components vector, x^(T) = [D_((k1,i))+c_((k1,k2))+D_((k2,j))]; t₀= start_time; dt = measurement_interval; While (!finished) { Whilewithin time interval (t₀, t₀ +dt) { Use a network monitoring tool tomeasure average round-trip delay, y₁, and delay-jitter, γ₁, between thepair of routers, π₁ = (R_((k1,i)), R_((k2,j))); } Estimate link-levelround-trip delay, D_((k1,i))+c_((k1,k2))+D_((k2,j)), as y_(1;) Estimatelink-level round-trip jitter, D_((k1,i))+c_((k1,k2))+D_((k2,j)), as γ₁;t₀ = t₀+dt } END “Sub_1_1”

EXAMPLE SCENARIOS

Three examples are now considered to illustrate the processes describedabove.

Example 1

Consider the enterprise network depicted in FIGS. 10A and 10B. Let thepair of routers be (R_(1,1)), R_((4,1))). Because n₁=3 and n₄=2, sub_2_2is applicable. As a result,Ω={π₁, π₂, π₃, π₄, π₅}  (98a)whereπ₁=(R _((1,1)) , R _((1,2)))  (98b)π₂=(R _((1,1)) , R _((4,1)))  (98c)π₃=(R _((4,1)) , R _((1,2)))  (98d)π₄=(R _((4,1)) , R _((4,2)))  (98e)π₅=(R _((1,1)) , R _((4,2)))  (98f)Further, the delay-components vector, x^(T)=[D_((1,1)), D_((1,2)),c_((1,4)), D_((4,1)), D_((4,2))]. An estimate of link-level round-tripdelay vector, x, may be computed using

[δ_(D(1,1)) ] [ 0.5 0.5 −0.5 0 0 ] [y₁] (99a) [δ_(D(1,2)) ] [ 0.5 −0.50.5 0 0 ] [y₂] (99b) [δ_(c(1,4)) ] = [ −0.5 0 0.5 −0.5 0.5 ] [y₃] (99c)[δ_(D(4,1)) ] [ 0 0.5 0 0.5 −0.5 ] [y₄] (99d) [δ_(D(4,2)) ] [ 0 −0.5 00.5 0.5 ] [y₅] (99e)An estimate of link-level round-trip jitter vector, x, may be computedusing

[σ² _(D(1,1))] [ 0.25 0.25 0.25 0 0 ] [γ₁ ²] (100a) [σ² _(D(1,2))] [0.25 0.25 0.25 0 0 ] [γ₂ ²] (100b) [σ² _(c(1,4))] = [ 0.25 0 0.25 0.250.25 ] [γ₃ ²] (100c) [σ² _(D(4,1))] [ 0 0.25 0 0.25 0.25 ] [γ₄ ²] (100d)[σ² _(D(4,2))] [ 0 0.25 0 0.25 0.25 ] [γ₅ ²] (100e)

Example 2

Letting the pair of routers be R_((1,1)), R_(2,1))), and because n₁=3and n₂=1, sub_2_1 is applicable. As a result, in a more compact formthan that just given,Ω={π₁=(R _((1,1)) , R _((1,2))), π₂=(R _((1,1)) , R _((2,1))), π₃=(R_((2,1)) , R _((1,2)))}  (101)The delay-components vector, x^(T)=[D_((1,1)), D_((1,2)),c_((1,4))+D_((2,1))]. An estimate of link-level round-trip delay vector,x, may be computed using

[δ_(D(1,1)) ] [ 0.5 0.5 −0.5 ] [y₁] (102a) [δ_(D(1,2)) ] = [ 0.5 −0.50.5 ] [y₂] (102b) [δ_(c(1,4)+(D(2,1)) ] [ −0.5 0.5 0.5 ] [y₃] (102c)An estimate of link-level round-trip jitter vector, x, may be computedusing

[σ² _(D(1,1)) ] [ 0.25 0.25 0.25 ] [γ₁ ²] (103a) [σ² _(D(1,2)) ] = [0.25 0.25 0.25 ] [γ₂ ²] (103b) [σ² _(c(1,2)+D(2,1)) ] [ 0.25 0.25 0.25] [γ₃ ²] (103c)It will be noted that it is not possible to apportion the delay orjitter between C_((1,2)) and D_((2,1)) separately since the number ofrouters in the second cluster is only 1.

Example 3

Let the pair of routers be (R_((3,1)), R_((2,1))). Because n₃=1 andn₂=1, sub_1_1 is applicable. Because Ω={π₁=(R_((3,1)), R_((2,1)))}, noestimate of individual delay components is available.

V. Estimating Delay and Jitter in an Enterprise Network Coupled to aBackbone Having an Unknown Topology Using a Large Number of Router Pairs

A method according to one embodiment of the present invention forcomputing the measurements set, Ω, for a given requirements set, Φ, hasbeen described. If the size of the set Φ is P, then the method involvesfinding a maximal set of independent vectors from a corresponding set ofP row vectors. Such a computation may be unreasonably difficult if P islarge (e.g., 10000 or more). Therefore, a method according to anembodiment of the present invention is now described that allows themeasurements set to be directly obtained. Measurements of delay andjitter between pairs of routers in this set may then be used to estimatedelay and jitter between any set of pairs of routers and specified by Φ.

FIG. 15 is a flow diagram illustrating the operations performed inestimating delay/jitter between router pairs in a specified requirementsset based on delay/jitter measurements between pairs of routers in ameasurements set. This measurement set is directly obtained once thecollection of routers that are covered by the requirements set isdetermined, and a model of the backbone network is computed. The processbegins with the specification of the requirement set as a collection ofrouter pairs (step 1500). Next, the collection of enterprise routers forwhich delay and jitter estimates are required is identified (step 1510).The process of “form_clusters” (e.g., as previously described) is thenused to:

-   -   1. determine a model for the enterprise network as one        consisting of clusters of routers (step 1520), and to    -   2. re-label the enterprise routers, as well as the requirements        set (steps 1530 and 1540).

Since a value w(k1, k2)=1 indicates that a delay over the correspondingvirtual link of the backbone network is to be estimated, the matrix w iszeroed out because no such determinations have as yet been recorded inthe matrix w (step 1545). Next, elements of the matrix signifyinginterconnection between clusters are set to one if the delay experiencedover the corresponding backbone link needs to be estimated (step 1550).The vector containing delay components is then identified (step 1555).

At this juncture, the process that computes the measurements sets iscalled, in order to identify the measurements set and to create a recordof equations that allows the estimation of delay and jitter (1560).Next, the start time and measurement interval are set (steps 1565 and1570). While further estimations of delay and jitter remain to be made(step 1575), the following steps are carried out. First, the averageround-trip delay/jitter for every router pair is measured during thetime interval (step 1580). From these measurements, delay and jitter foreach delay component is estimated (step 1585). Also estimated is theround-trip and round-jitter for each router pair in Φ (step 1590). Iffurther measurements are desired, the start time is incremented and theprocess repeated (step 1595).

The preceding method for computing the measurements set for a givenrequirements set when a maximal set of independent vectors from acorresponding set of row vectors is to be found is now outlined using aformal representation.

BEGIN /* Specify the requirements set as a collection of router pairs *//* Note that the following router pairs may not be, for example,(R_((k1,i)), ρ_(k2)) */ /* Further, without loss of generality, notethat it can be assumed that i < j */ /* (This is valid becauseround-trip delay and jitter are examined, */ /* and so π_(κ) = (R_(i),R_(j))} = (R_(j), R_(i)) can be assumed) */ Φ = {π_(κ) = (R_(i), R_(j)),κ = 1, 2, . . . , P}; /* From the given requirements set, Φ, identify *//* the collection of enterprise routers, , for which */ /* delay andjitter estimates are required */ = ∪ R_(i); /* such that for some R_(j),(R_(i), R_(j)) ∈ Φ or (R_(j), R_(i)) ∈ Φ */ /* Given , use“form_clusters” to: */ /* (1) Determine a model for the enterprisenetwork as one having */ /* clusters of routers, C_(k) = {R_((k,i)), i =1, 2, . . . , n_(k)}, k = 1, 2, . . . , m, */ /* where n_(k)= |C_(k)| *//* (2) Re-label the requirements set, Φ = {π_(κ) = (R_((k1,i)),R_((k2,j)))} */ form_clusters ( ); /* Re-order elements of Φ */ For allκ = 1 to P do { Relabel π_(k) = (R_(i), R_(j)) as π_(k) = (R_((k1,i), R)_((k2,j))) } /* Identify interconnection between clusters using virtualservice-provider links. This is specified as a m × m matrix, ω(k1, k2).Note that ω(k1,k2) = 1 indicates that delay over link λ_((k1,k2)) isrequired to be estimated, by definition. */ /* Zero out the matrix ω */For all k1 = 1, 2, . . . , m do { For all k2 = 1, 2, . . . m do{ω(k1,k2) = 0 } } /* Set matrix elements to one if the delay over thecorresponding backbone link */ /* needs to be estimated */ For all{π_(κ) = (R_((k1,i)), R_((k2,j)))} ∈ Φ do { ω(k1,k2) = 1 } /* Identifythe vector of delay components */ {D_((k,i)), k = 1, 2, . . . , m, i =1, 2, . . . n_(k)} ∪ {c_((k1,k2)), k1, k2 = 1, 2, . . . , m, ω(k1,k2) =1} /* Call compute_measurement_set to: */ /* Identify the measurementsset, Ω = {π_(κ) = (R_((k1,i)), R_((k2,j)), κ = 1, 2, . . . , Q} */ /*Create record of eqns allowing est. of delay/jitter) of D(k1,i) orc(k1,k2): */ /* est_delay( ) = function(y_(κ), . . . ,est_delay(D_((k1,i))), . . . ) */ /* est_jitter( ) = function(γ_(κ), . .. , est_jitter(D_((k1,i))), . . . ) */ compute_measurement_set( ); t₀ =start_time; dt = measurement_interval; While (!finished) { Measure theaverage round-trip delay, y_(κ), and delay-jitter, γ_(κ), over timeinterval [t₀, t₀+dt] for every pair of routers, π_(κ) = (R_((k1,i)),R_((k2,j)) ∈ Ω, using a network monitoring tool; /* Estimate delay andjitter for each delay component */ For all k = 1, 2, . . . , m do { Forall i = 1, 2, . . . , n_(k) do { evaluate est_delay(D_((k,i))); evaluateest_jitter(D_((k,i))) } } For all k1 = 1, 2, . . . , m do { For all k2 =k1+1, k2+2, . . . , m do { if ω(k1,k2) = 1 then { evaluateest_delay(c_((k1,k2))); evaluate est_jitter(c_((k1,k2))) } } } /* Foreach router pair in Φ, estimate round-trip delay, Δ_(κ), and round-tripjitter, σ_(κ) */ For each π_(κ) = (R_((k1,i)), R_((k2,j))) ∈ Φ do { If(((R_((k1,i)), R(_(k2,j))) = π′_(k1))∈ Ω) then { Δ_(κ) = y_(κ1); σ_(κ) =γ_(κ1) } else { if k1 = k2 then { Δ_(κ) = est_delay(D_((k1,i))) +est_delay(D_((k2,j))); σ_(κ) = √ (est_jitter(D_((k1,i)))² +est_jitter(D_((k2,j)))²) } else { Δ_(κ) = est_delay(D_((k1,i))) +est_delay(c_((k1,k2))) + est_Delay(D_((k2,j))); σ_(κ) = √ (est_jitter(D_((k1,i)))² + est_jitter(c_((k1,k2)))² +est_jitter(D_((k2,j)))²) } }; } t₀ = t₀+dt } END

FIG. 16 is a flow diagram illustrating the operations performed incomputing the measurements set needed in the preceding method. It isassumed that the given network has been described in terms of clustersand an interconnection matrix, using a method of clustering such as thatdiscussed previously. The process begins with the initialization of themeasurements set (step 1600). For all clusters (defined by k1=1, . . . ,m) (step 1604), the following steps are performed.

First, the size of the given cluster is examined (step 1606). If thegiven cluster has two or more enterprise routers, a second cluster(identified by k2) is identified, such that delay/jitter over thebackbone link (virtual link) between the two clusters (or,alternatively, backbone routers) must be obtained (i.e., such thatω(k1,k2)=1 or ω(k2,k1)=1) (step 1608). Having identified this secondcluster, the procedure identifies three specific routers and adds thecorresponding pairs of routers to the measurements set (steps 1610, 1612and 1614). Expressions representing estimated delay over linksconnecting enterprise routers or over the virtual backbone links arethen recorded (step 1616).

The rest of the enterprise routers in the given cluster are thenanalyzed (step 1620). First, a router pair corresponding every third,fourth, . . . , router in the cluster current is added to themeasurements set (step 1622). Next, expressions representing estimateddelay and jitter between associated backbone and enterprise routers isrecorded (steps 1624 and 1626). This completes the scenario in which thegiven cluster's size is greater than or equal to two.

If the given cluster's size is equal to one (step 1606), the followingactions are performed. A delay estimate of zero is recorded, indicatingthat such an estimate is not available (step 1628). In similar fashion,a jitter estimate of zero is recorded, likewise indicating that such anestimate is not available (step 1630). This completes this portion ofthe analysis.

Once the addition of router pairs to the measurements set and therecording of delay and jitter expressions are completed, theinter-cluster router pairs (and associated delay and jitter) aredetermined. For pairs of clusters consisting of a first cluster (one ofall clusters) and a second cluster (one of all remaining clusters) thefollowing actions are performed (steps 1632 and 1634). If delay betweenbackbone routers corresponding to the two clusters must be estimated(step 1636), then a pair of enterprise routers representing the twoclusters is added to the measurements set (step 1638). Expressionsrepresenting the estimated delay (step 1640) and estimated jitter (step1642) associated with the given pair of backbone routers are recorded.Finally, the size of the measurements set is stored (step 1644). Thiscompletes computation of the measurements set.

The preceding method for computing the measurements set is now outlinedusing a formal representation. For a given network described in terms ofclusters:C _(k) {R _((k,1)) , i=1, 2, . . . , n _(k) }, k=1, 2, . . . , m, wheren_(k) =|C _(k)|  (104)and an interconnection matrix:ω(k 1,k 2), k 1, k 2=1, 2, . . . , m  (105)the measurements set can be computed as follows:

BEGIN “compute_measurements_set” Ω = { } /* Ω = measurements set */ κ =0 /* κ identifies the measurements, y_(κ) ₂ */ For all k1 = 1, 2, . . ., m do { If n_(k1) ≧ then { /* It is assumed that such a k2 exists */identify any k2, such that ω(k1,k2) = 1 or ω(k2,k2) = 1; κ1 = κ +1;add_to__set(π_(κ1) = (R_((k1,1)), R_((k1,2))), Ω); κ2 = κ +2;add_to_set(π_(κ2) = (R_((k1,1)), R_((k2,1))), Ω); κ3 = κ +3;add_to_set(π_(κ3) = (R_((k2,1)), R(_(k1,2))), Ω); κ = κ +3; /* Onemethod to record an expression is to use a binary evaluation tree */rec_eqn[est_delay(D_((k1,1))) = 0.5 y_(κ1) + 0.5 y_(κ2) − 0.5 y_(κ3)];rec_eqn[est_delay(D_((k1,2))) = 0.5 y_(κ1) − 0.5 y_(κ2) + 0.5 y_(κ3)];/* An estimate of jitter is similarly derived */rec_eqn[est_jitter(D_((k1,1))) = √ (0.25 γ² _(κ1) + 0.25 γ² _(κ2) + 0.25γ² _(κ3))]; rec_eqn[est_jitter(D_((k1,2))) = √(0.25 γ² _(κ1) + 0.25 γ²_(κ2) + 0.25 γ² _(κ3))]; For all i = 3, 4, . . . , n_(k1) do { κ = κ +1;add_to_set(π_(κ) = (R_((k1,1)), R_((k1,i))), Ω);rec_eqn[est_delay(D_((k1,i))) = y_(κ) − est_delay(D_((k1,1)))];rec_eqn[est_jitter(D_((k1,i))) = √(y² _(κ) + est_jitter(D_((k1,1)))²); }} else { /* A delay estimate of 0 signifies that estimate is notavailable */ rec_eqn[est_delay(D_((k1,1))) = 0];rec_eqn[est_jitter(D_((k1,1))) = 0]; } } For all k1 = 1, 2, . . . , m do{ For all k2 = k1+1, k1+2, . . . , m do { if ω(k1,k2) = 1 then { κ = κ+1; add_to_set(π_(κ) = (R_((k1,1)), R_((k2,1))), Ω);rec_eqn[est_delay(c_((k1,k2))) = y_(κ) − est_delay(D_((k1,1))) −est_delay(D_((k2,1)))]; rec_eqn[est_jitter(c_((k1,k2))) = √(γ² _(κ)+est_jitter(D_((k1,1)))² +est_jitter(D_((k2,1)))²)] } } } Q = k /* Storethe size of the measurements set */ END “compute_measurements_set”

Using the foregoing method, an estimate of all (or nearly all) delaycomponents is available, depending on the topologies involved. Thus, itis possible to apportion end-to-end delay among different links thatconstitute the given path. It will be noted that the measurements set,Ω, may or may not be a sub-set of the requirements set, Φ. The totalnumber of measurements required is at most:

 (n ₁−1+n ₂−1+ . . . n _(m)−1)+2m+m(m−1)/2  (105)

This bound is achieved when every cluster has at least 2 routers, anddelay/jitter estimates are required between every pair of clusters.Moreover, the above method suggests that delay and jitter measurementsbetween π_(κ)=(R_(k1,1)), R_((k2,1))) may be made more than once. Infact, this may be rectified during implementation by checking whether adelay/jitter measurement between π_(κ)=(R_((k1,1)), R_((k2,1))) isalready available. In such a case, the pre-existing measurement is used.Exemplary Enterprise Network

As an example, the enterprise network of FIG. 10B may again beconsidered, with the requirements set: $\quad\begin{matrix}{\Phi = \left\{ {\left( {R_{({1,1})},R_{({2,1})}} \right),\left( {R_{({1,2})},R_{({2,1})}} \right),\left( {R_{({1,3})},R_{({2,1})}} \right),} \right.} \\{\left( {R_{({1,1})},R_{({3,1})}} \right),\left( {R_{({1,2})},R_{({3,1})}} \right),\left( {R_{({1,3})},R_{({3,1})}} \right),} \\{\left( {R_{({1,1})},R_{({4,1})}} \right),\left( {R_{({1,2})},R_{({4,1})}} \right),\left( {R_{({1,3})},R_{({4,1})}} \right),} \\\left. {\left( {R_{({1,1})},R_{({4,2})}} \right),\left( {R_{({1,2})},R_{({4,2})}} \right),\left( {R_{({1,3})},R_{({4,2})}} \right)} \right\}\end{matrix}$The collection of routers is ={R_((1,1)), R_((1,2)), R_((1,3)),R_((2,1)), R_((3,1)), R_((4,1)), R_((4,2))}. Further, the matrix, ω is

K2 = 1 K2 = 2 K2 = 3 K2 = 4 K1 = 1 1 1 1 K1 = 2 — K1 = 3 — K1 = 4 —The delay-component vector is given as $\quad\begin{matrix}{x^{T} = \begin{bmatrix}D_{({1,1})} & \quad\end{bmatrix}} \\{\begin{bmatrix}D_{({1,2})} & \quad\end{bmatrix}} \\{\begin{bmatrix}D_{({1,3})} & \quad\end{bmatrix}} \\{\begin{bmatrix}D_{({2,1})} & \quad\end{bmatrix}} \\{\begin{bmatrix}D_{({3,1})} & \quad\end{bmatrix}} \\{\begin{bmatrix}D_{({4,1})} & \quad\end{bmatrix}} \\{\begin{bmatrix}D_{({4,2})} & \quad\end{bmatrix}} \\{\begin{bmatrix}c_{({1,2})} & \quad\end{bmatrix}} \\{\begin{bmatrix}c_{({1,3})} & \quad\end{bmatrix}} \\{\begin{bmatrix}c_{({1,4})} & \quad\end{bmatrix}}\end{matrix}$

Using the subroutine “compute_measurements_set,” the measurements set isobtained as follows:

-   (i) Ω={ }-   (ii) Let k1=1: because n₁=3, Ω←Ω∪{(R_((1,1)), R_((1,2))),    (R_((1,1)), R_((2,1))), (R_((2,1)), R_((1,2))), (R_((1,1)),    R_((1,3)))}-   (iii) Let k1=2: because n₂=1, there is no change in Ω.-   (iv) Let k1=3: because n₃=1, there is no change in Ω.-   (v) Let k1=4, because n₄=2, Ω=Ω∪{(R_((4,1)), R_((4,2))), (R_((4,1)),    R_((1,1))), (R_((1,1)), R_((4,2)))}-   (vi) Let k1=1, k2=2, because ω(k1,k2)=1, Ω=Ω∪{(R_((1,1)),    R_((2,1)))}-   (vii) Let k1=1, k2=3, because ω(k1,k2)=1, Ω=Ω∪{(R_((1,1)),    R_((3,1)))}-   (viii) Let k1=1, k2=4, because ω(k1,k2)=1, Ω=Ω∪{(R_((1,1)),    R_((4,1)))}.    As a result, $\quad\begin{matrix}    {\Omega = \left\{ {\left( {R_{({1,1})},R_{({1,2})}} \right),\left( {R_{({1,1})},R_{({2,1})}} \right),\left( {R_{({2,1})},R_{({1,2})}} \right),\left( {R_{({1,1})},R_{({1,3})}} \right),} \right.} \\    \left. {\left( {R_{({4,1})},R_{({4,2})}} \right),\left( {R_{({4,1})},R_{({1,1})}} \right),\left( {R_{({1,1})},R_{({4,2})}} \right),\left( {R_{({1,1})},R_{({3,1})}} \right)} \right\}    \end{matrix}$    and the estimates of various delay components may be obtained from    the corresponding measurements, y_(i), i=1, 2, . . . , 8, as    follows:    -   est_delay(D_((1,1)))=0.5y₁+0.5y₂−0.5y₃    -   est_delay(D_((1,2)))=0.5y₁−0.5y₂+0.5y₃    -   est_delay(D_((1,3)))=y₄−est_delay(D_((1,1)))    -   est_delay(D_((2,1)))=0 /* (no estimate is available) */    -   est_delay(D_((3,1)))=0 /* (no estimate is available) */    -   est_delay(D_((4,1)))=0.5y₅+0.5y₆−0.5y₇    -   est_delay(D_((4,2)))=0.5y₅−0.5y₆+0.5y₇    -   est_delay(c_((1,2)))=y₂−est_delay(D_((1,1)))−est_delay(D_((2,1)))]    -   est_delay(c_((1,3)))=y₈−est_delay(D_((1,1)))−est_delay(D_((3,1)))]    -   est_delay(c_((1,4)))=y₆−est_delay(D_((1,1)))−est_delay(D_((4,1)))]

These estimates may now be combined to estimate the delay between anypair of enterprise routers. For instance, the delay between R_((1,3))and R_((4,2)) may be estimated to be

-   -   est_delay(D_((1,3)))+est_delay(c_((1,4)))+est_delay(D_((4,2)))        The equations for estimating jitter are similar.

While particular embodiments of the present invention have been shownand described, it will be obvious to those skilled in the art that,based upon the teachings herein changes and modifications may be madewithout departing from this invention and its broader aspects and,therefore, the appended claims are to encompass within their scope allsuch changes and modifications as are within the true spirit and scopeof this invention. Furthermore, it is to be understood that theinvention is solely defined by the appended claims.

1. A method comprising: identifying a first set of network element pairsfrom a plurality of network elements, each of said network elementsbeing coupled to another of said network elements by at least one of aplurality of links, by: identifying a first plurality of network elementpairs from said network elements, wherein each of said first pluralityof network element pairs corresponds to a one of said links, andidentifying a network element pair from said network elements, whereineach of a first and a second network element of said network elementpair is adjacent to a one of said network elements, a path between afirst and a second network element of said network element paircomprises said one of said network elements, and said first set ofnetwork element pairs comprises said first plurality of network elementpairs and said network element pair.
 2. The method of claim 1, furthercomprising: identifying another network element pair from said networkelements, wherein every path between a first network element and asecond network element of said another network element pair, if any,comprises a plurality of said network elements.
 3. The method of claim2, further comprising: measuring a measured network performance metricover said first set of network element pairs; and using said measurednetwork performance metric to estimate an estimated network performancemetric over said another network element pair.
 4. The method of claim 3,wherein a second set of network element pairs comprises said anothernetwork element pair.
 5. The method of claim 4, wherein said usingcomprises: computing said estimated network performance metric betweensaid first network element and said second network element of saidanother network element pair.
 6. The method of claim 5, furthercomprising: identifying pairs of said network elements as being in saidsecond set of network element pairs; and identifying pairs of saidnetwork elements in said second set of network element pairs as being insaid first set of network element pairs.
 7. The method of claim 6,further comprising: generating a first matrix using said second set ofnetwork element pairs, wherein each row in said first matrix correspondsto a corresponding network element pair in said second set of networkelement pairs, said first matrix comprises independent rows andnon-independent rows, said first set of network element pairs containsindependent network element pairs in said second set of network elementpairs, and each one of said independent pairs of network elementcorresponds to a one of said independent rows of said first matrix. 8.The method of claim 7, wherein said measured network performance metricis measured between a first network element and a second network elementof each network element pair in said first set of network element pairs.9. The method of claim 7, said method further comprising: computing anumber, wherein said number is equal to a rank of said first matrix;determining if a first said number of rows of said first matrix areindependent; and if said first said number of said rows of said firstmatrix are not independent, re-arranging said rows of said first matrixsuch that said first said number of said rows of said first matrix areindependent.
 10. The method of claim 9, further comprising: identifyinga maximal set of independent rows of said first matrix based on saidnumber.
 11. The method of claim 9, wherein said rearranging furthercomprises: re-arranging said pairs of said network elements in saidsecond set of network element pairs such that said correspondencebetween each row of said first matrix and said corresponding networkelement pair in said second set of network element pairs is maintained.12. The method of claim 11, wherein said forming said first set ofnetwork element pairs comprises: copying a first said number of pairs ofsaid network elements in said second set of network element pairs intosaid first set of network element pairs.
 13. The method of claim 7,wherein said network element pair in said second set of network elementpairs is a remaining network element pair in said second set of networkelement pairs, and said remaining network element pair corresponds to anon-independent row of said first matrix.
 14. The method of claim 8,wherein said computing said estimated network performance metric betweensaid first network element and said second network element of saidremaining network element pair comprises: forming a second matrix,wherein each row of said second matrix corresponds to a correspondingone of said non-independent rows of said first matrix, and said each rowof said second matrix is such that said corresponding one of saidnon-independent rows of said first matrix can be expressed in terms ofsaid independent rows using said each row of said second matrix;organizing said measured network performance metrics into a vector; andcomputing said estimated network performance metric between said firstnetwork element and said second network element of said remainingnetwork element pair by multiplying said vector by a row of said secondmatrix corresponding to said remaining network element pair.
 15. Themethod of claim 8, wherein said computing said estimated networkperformance metric between said first network element and said secondnetwork element of said remaining network element pair comprises:creating a vector equivalent to said non-independent row of said firstmatrix by combining a plurality of said independent rows of said firstmatrix; and computing said estimated network performance metric bycombining a measured network performance metric of each network elementpair of said first set of network element pairs corresponding to one ofsaid plurality of said independent rows of said first matrix.
 16. Amethod comprising: identifying a first set of router pairs from aplurality of network elements, each of said network elements beingcoupled to another of said network elements by at least one of aplurality of links; forming a second set of network element pairs byidentifying a first plurality of network element pairs from said firstset of router pairs, wherein each of said first plurality of networkelement pairs corresponds to a one of said links, and identifying anetwork element pair from said first set of router pairs, wherein eachof a first and a second network element of said network element pair isadjacent to a one of said network elements, a path between a first and asecond network element of said network element pair comprises said oneof said network elements, and said first set of network element pairscomprises said first plurality of network element pairs and said networkelement pair; and identifying another network element pair from saidnetwork elements, wherein every path between a first network element anda second network element of said another network element pair, if any,comprises a plurality of said network elements, and said second set ofnetwork element pairs comprises said first set of network element pairsand said another network element pair.
 17. The method of claim 16,further comprising: generating a first matrix from said second set ofnetwork element pairs, wherein each row in said first matrix correspondsto a corresponding network element pair in said second set of networkelement pairs.
 18. The method of claim 17, further comprising: measuringa measured network performance metric between a first network elementand a second network element of each network element pair in said secondset of network element pairs.
 19. The method of claim 18, furthercomprising: generating a second matrix from said second set of networkelement pairs, wherein each row in said second matrix stores a measurednetwork performance metric corresponding to a network element pair insaid second set of network element pairs.
 20. The method of claim 19,further comprising: computing an estimated network performance metricbetween a first network element and a second network element of saidanother network element pair using at least one of said measured networkperformance metrics.
 21. The method of claim 19, wherein said first setof network element pairs is a requirements set.
 22. The method of claim21, wherein said second set of network element pairs is a measurementsset.
 23. The method of claim 22, wherein each one of said networkelements is a router.
 24. The method of claim 19, wherein said firstmatrix is a matrix H, and said second matrix is a matrix x.
 25. Themethod of claim 24, further comprising: re-arranging columns of saidmatrix H such that a column corresponding to a delay through said one ofsaid elements is positioned after all columns corresponding to a delayover said one of said links; and re-arranging components of said matrixx to maintain correspondence between said columns of said matrix H andsaid components of said matrix x.
 26. The method of claim 25, furthercomprising: re-arranging said matrix H such that said matrix H appearsas $\quad{\begin{matrix}{H = \begin{bmatrix}\begin{matrix}\quad & I\end{matrix} & \begin{matrix}{H12} & {H13}\end{matrix}\end{bmatrix}} \\{\begin{bmatrix}\begin{matrix}\quad & {H21}\end{matrix} & \begin{matrix}{H22} & {H23}\end{matrix}\end{bmatrix}}\end{matrix}.}$
 27. The method of claim 26, further comprising:constructing a matrix B and a matrix C from said matrix H such that saidmatrix B appears as $\quad{\begin{matrix}{B = \begin{bmatrix}\begin{matrix}\quad & I\end{matrix} & \begin{matrix}{{{- 1}/2}\quad{H12}} & \quad\end{matrix}\end{bmatrix}} \\{\begin{bmatrix}\begin{matrix}\quad & 0\end{matrix} & \begin{matrix}{{1/2}\quad I} & \quad\end{matrix}\end{bmatrix}}\end{matrix},}$  and said matrix C appears as $\quad{\begin{matrix}{C = \begin{bmatrix}\begin{matrix}\quad & I\end{matrix} & \begin{matrix}0 & \quad\end{matrix}\end{bmatrix}} \\{\begin{bmatrix}\begin{matrix}\quad & {H21}\end{matrix} & \begin{matrix}{- I} & \quad\end{matrix}\end{bmatrix}}\end{matrix}.}$
 28. The method of claim 26, further comprising:computing a matrix product D=BC.
 29. The method of claim 28, furthercomprising: construct a matrix F from said matrix product D.
 30. Acomputer program product encoded in a computer readable media which whenexecuted causes a computer to perform a method comprising: identifying afirst set of network element pairs from a plurality of network elements,each of said network elements being coupled to another of said networkelements by at least one of a plurality of links, by: identifying afirst plurality of network element pairs from said network elements,wherein each of said first plurality of network element pairscorresponds to a one of said links, and identifying a network elementpair from said network elements, wherein each of a first and a secondnetwork element of said network element pair is adjacent to a one ofsaid network elements, a path between a first and a second networkelement of said network element pair comprises said one of said networkelements, and said first set of network element pairs comprises saidfirst plurality of network element pairs and said network element pair.31. The computer program product of claim 30, said method furthercomprising: identifying another network element pair from said networkelements, wherein every path between a first network element and asecond network element of said another network element pair, if any,comprises a plurality of said network elements.
 32. The computer programproduct of claim 31, said method further comprising: measuring ameasured network performance metric over said first set of networkelement pairs; and using said measured network performance metric toestimate an estimated network performance metric over said anothernetwork element pair.
 33. The computer program product of claim 32,wherein a second set of network element pairs comprises said anothernetwork element pair.
 34. The computer program product of claim 33,wherein said using comprises: computing said estimated networkperformance metric between said first network element and said secondnetwork element of said another network element pair.
 35. The computerprogram product of claim 34, said method further comprising: identifyingpairs of said network elements as being in said second set of networkelement pairs; and identifying pairs of said network elements in saidsecond set of network element pairs as being in said first set ofnetwork element pairs.
 36. The computer program product of claim 35,said method further comprising: generating a first matrix using saidsecond set of network element pairs, wherein each row in said firstmatrix corresponds to a corresponding network element pair in saidsecond set of network element pairs, said first matrix comprisesindependent rows and non-independent rows, said first set of networkelement pairs contains independent network element pairs in said secondset of network element pairs, and each one of said independent pairs ofnetwork element corresponds to a one of said independent rows of saidfirst matrix.
 37. The computer program product of claim 36, wherein saidmeasured network performance metric is measured between a first networkelement and a second network element of each network element pair insaid first set of network element pairs.
 38. The computer programproduct of claim 36, said method further comprising: computing a number,wherein said number is equal to a rank of said first matrix; determiningif a first said number of rows of said first matrix are independent; andif said first said number of said rows of said first matrix are notindependent, re-arranging said rows of said first matrix such that saidfirst said number of said rows of said first matrix are independent. 39.The computer program product of claim 38, said method furthercomprising: identifying a maximal set of independent rows of said firstmatrix based on said number.
 40. The computer program product of claim38, wherein said rearranging further comprises: re-arranging said pairsof said network elements in said second set of network element pairssuch that said correspondence between each row of said first matrix andsaid corresponding network element pair in said second set of networkelement pairs is maintained.
 41. The computer program product of claim40, wherein said forming said first set of network element pairscomprises: copying a first said number of pairs of said network elementsin said second set of network element pairs into said first set ofnetwork element pairs.
 42. The computer program product of claim 36,wherein said network element pair in said second set of network elementpairs is a remaining network element pair in said second set of networkelement pairs, and said remaining network element pair corresponds to anon-independent row of said first matrix.
 43. The computer programproduct of claim 37, wherein said computing said estimated networkperformance metric between said first network element and said secondnetwork element of said remaining network element pair comprises:forming a second matrix, wherein each row of said second matrixcorresponds to a corresponding one of said non-independent rows of saidfirst matrix, and said each row of said second matrix is such that saidcorresponding one of said non-independent rows of said first matrix canbe expressed in terms of said independent rows using said each row ofsaid second matrix; organizing said measured network performance metricsinto a vector; and computing said estimated network performance metricbetween said first network element and said second network element ofsaid remaining network element pair by multiplying said vector by a rowof said second matrix corresponding to said remaining network elementpair.
 44. The computer program product of claim 37, wherein saidcomputing said estimated network performance metric between said firstnetwork element and said second network element of said remainingnetwork element pair comprises: forming a second matrix, wherein eachrow of said second matrix corresponds to a corresponding one of saidnon-independent rows of said first matrix, and said each row of saidsecond matrix is such that said corresponding one of saidnon-independent rows of said first matrix can be expressed in terms ofsaid independent rows using said each row of said second matrix;organizing said measured network performance metrics into a vector; andcomputing said estimated network performance metric between said firstnetwork element and said second network element of said remainingnetwork element pair by multiplying said vector by a row of said secondmatrix corresponding to said remaining network element pair.
 45. Anapparatus comprising: means for identifying a first set of networkelement pairs from a plurality of network elements, each of said networkelements being coupled to another of said network elements by at leastone of a plurality of links, said means for identifying said first setof network element pairs comprising: means for identifying a firstplurality of network element pairs from said network elements, whereineach of said first plurality of network element pairs corresponds to aone of said links, and means for identifying a network element pair fromsaid network elements, wherein each of a first and a second networkelement of said network element pair is adjacent to a one of saidnetwork elements, a path between a first and a second network element ofsaid network element pair comprises said one of said network elements,and said first set of network element pairs comprises said firstplurality of network element pairs and said network element pair. 46.The apparatus of claim 45, further comprising: means for identifyinganother network element pair from said network elements, wherein everypath between a first network element and a second network element ofsaid another network element pair, if any, comprises a plurality of saidnetwork elements.
 47. The apparatus of claim 46, further comprising:means for measuring a measured network performance metric over saidfirst set of network element pairs; and means for using said measurednetwork performance metric to estimate an estimated network performancemetric over said another network element pair.
 48. The apparatus ofclaim 47, wherein a second set of network element pairs comprises saidanother network element pair.
 49. The apparatus of claim 48, whereinsaid means for using comprises: means for computing said estimatednetwork performance metric between said first network element and saidsecond network element of said another network element pair.
 50. Theapparatus of claim 49, further comprising: means for identifying pairsof said network elements as being in said second set of network elementpairs; and means for identifying pairs of said network elements in saidsecond set of network element pairs as being in said first set ofnetwork element pairs.
 51. The apparatus of claim 50, furthercomprising: means for generating a first matrix using said second set ofnetwork element pairs, wherein each row in said first matrix correspondsto a corresponding network element pair in said second set of networkelement pairs, said first matrix comprises independent rows andnon-independent rows, said first set of network element pairs containsindependent network element pairs in said second set of network elementpairs, and each one of said independent pairs of network elementcorresponds to a one of said independent rows of said first matrix. 52.A data processing system comprising: a plurality of network elements,each of said network elements being coupled to another of said networkelements by at least one of a plurality of links; and a computer systemconfigured to identify a first set of network element pairs from saidplurality of network elements by virtue of being configured to identifya first plurality of network element pairs from said network elements,wherein each of said first plurality of network element pairscorresponds to a one of said links, and identify a network element pairfrom said network elements, wherein each of a first and a second networkelement of said network element pair is adjacent to a one of saidnetwork elements, a path between a first and a second network element ofsaid network element pair comprises said one of said network elements,and said first set of network element pairs comprises said firstplurality of network element pairs and said network element pair.